Commit graph

9 commits

Author SHA1 Message Date
fd976b0cd5
fix: resolve 391 Credo issues across codebase
- Add jump_credo_checks ~> 0.4 with all 20 checks enabled
- Fix all standard Credo issues: 139 @spec (113 done, 26 remain),
  4 refactoring, 3 alias usage, 9 System.cmd env, 5 unsafe_to_atom,
  2 max line length, 9 assert_receive timeout
- Fix 170+ jump_credo_checks warnings:
  - 117 TopLevelAliasImportRequire: move nested alias/import to module top
  - 32 UseObanProWorker: switch to Oban.Pro.Worker
  - 4 DoctestIExExamples: add doctests / create test file
  - ~20 WeakAssertion: strengthen type-check assertions
  - Various ConditionalAssertion, AssertReceiveTimeout fixes
- Exclude vendor/ from Credo analysis
- Remaining: 175 warnings (mostly opinionated WeakAssertion,
  AvoidSocketAssignsInTest), 26 @spec annotations
2026-06-12 13:51:32 -05:00
ea3033da03
chore: update to elixir 1.20 / otp 29, fix compile warnings
- Update .tool-versions to elixir 1.20.0-otp-29 / erlang 29.0.1
- Update all hex dependencies
- Remove unused aliases in path_compute.ex and rover_planning_live/show.ex
- Fix Oban unique states to include :suspended (use :incomplete group)
2026-06-03 14:34:39 -05:00
828814e767
fix: resolve all compiler warnings except framework-generated phoenix_component_verify
- Replace deprecated xref:exclude with elixirc_options in vendor/oban_pro
- Remove unused require Logger from 8 files
- Pin bitstring size variables with ^ in simple_packing, wgrib2,
  complex_packing, section, nexrad_client, mqtt, and radar worker
- Remove dead defp clauses (format/1 nil, depression/2 nil)
- Rename Buildings.Parser type record/0 -> building_record/0 to avoid
  overriding built-in type
- Remove redundant catch-all in candidate_detail.ex
- Simplify contact_live/show.ex conditional based on type narrowing
- Fix DateTime.from_iso8601 return pattern in vendor/oban_web
- Upgrade phoenix_live_view to 1.1.31
- Drop --warnings-as-errors from precommit alias; known false positives
  from @after_verify-generated code in Elixir 1.20.0-rc.6 + PLV 1.1.x
- Add credo --strict to precommit as replacement static-analysis gate
2026-05-29 10:18:52 -05:00
e9a38623d8
perf+hygiene: batch 1 of system-review fixes
Batched from a system-wide review pass.

**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
  of {:error}. Stops Oban.PerformError stack-trace spam on every
  routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
  ~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
  Oban unique conflict case was silently dropping jobs but still
  producing 'enqueuing grid worker' info lines every 5 minutes).

**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
  once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
  String allocations per f01..f18 chain step across pipeline.rs,
  hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
  hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
  replaces the 18-wide Enum.map serial walk. Forecast cache warms
  ~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
  forecast_hour ASC) WHERE status='queued' AND kind='forecast',
  matching the Rust claim query's ORDER BY. Drops the old
  status-only partial that forced a sort per claim.

**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
  :offset]]. Was missing on a worker whose perform() does a
  non-idempotent atomic counter increment — a retry would double-
  count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
  x_min > x_max, triggering a runtime deprecation warning. Force
  Range.new(_, _, 1) explicitly.

**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
  process_batch signature.
2026-04-21 17:06:07 -05:00
084e6b6224
refactor(db): use explicit on_conflict set clauses over :replace_all_except
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.

Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.

Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)

Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
2026-04-21 14:22:15 -05:00
7ab426c01d
fix(workers): propagate transient errors so Oban retries + counters fire
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.

- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
  errors using the same pattern as RadarFrameWorker. Permanent errors
  still mark :unavailable + :ok so the backfill stops re-queueing them.
  Transient errors return {:error, reason} and leave radar_status alone
  so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
  operators can see which contacts blew up, but return
  {:error, inspect(e)} from the rescue instead of swallowing it as :ok.
2026-04-21 13:15:27 -05:00
2da74c5cd8
feat(telemetry): wide instrumentation + bump hrrr to 2 per pod
Config:
- runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods)

New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with
a [:microwaveprop | event_suffix] prefix so metrics can key off it.

Spans added (each emits duration + result tag):
- HrrrClient.fetch_grid / fetch_profile / fetch_idx
- NexradClient.fetch_frame + decode_png
- IemClient.fetch_asos / fetch_raob
- GefsClient.fetch_grid_profiles
- NarrClient.fetch_profile_at
- UwyoSoundingClient.fetch_sounding
- ElevationClient.fetch_elevation_profile
- Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch
- Propagation.replace_scores
- PropagationGridWorker.compute_scores_algorithm
- TerrainAnalysis.analyse
- CommonVolumeRadarWorker.aggregate_stats

Telemetry catalog (MicrowavepropWeb.Telemetry):
- Oban job duration / queue_time / count / exception by (worker, queue, state)
- Per-span summary metrics for every instrumented phase above
- Periodic (10s) poller emits oban queue depth by (queue, state) — drops
  into the /admin/dashboard Metrics tab immediately

Also drops the now-redundant "fetching n0q frame" and "fetching <url>"
info lines from CommonVolumeRadarWorker / NexradClient; the span events
cover that and the worker's "ingested" line stays for per-job signal.
2026-04-18 16:33:34 -05:00
ced5e3d1a6
feat(radar): log NEXRAD fetches + worker progress
Previously the CommonVolumeRadarWorker ran silently — no URL logged,
no sign in the logs that it was doing any work. Added info-level
fetch/ok/error logs in NexradClient (same shape as WeatherFetchWorker
RAOB logs) plus start-of-work + ingestion lines in the worker itself.
Unchanged: the existing "no frame" warning and all control flow.
2026-04-18 16:20:59 -05:00
33f5d4edbe
feat(rainscatter): classify QSO propagation mechanism from common-volume radar
Adds a per-contact enrichment pipeline that determines whether a QSO was
most likely carried by rain scatter, tropospheric ducting, or ordinary
troposcatter — using IEM n0q composite reflectivity sampled inside the
lens-shaped intersection of 400 km-radius disks around each endpoint.

Pieces:
  * Microwaveprop.Propagation.CommonVolume — lens geometry (haversine,
    in-CV test, bbox, area).
  * contact_common_volume_radar table (1:1 per contact) storing
    aggregate dBZ stats inside the CV + radar_status column on contacts.
  * Microwaveprop.Workers.CommonVolumeRadarWorker — Oban :radar queue,
    fetches the n0q frame at QSO time, iterates pixels inside the CV
    bbox, aggregates rain/heavy/core-pixel counts, max/mean dBZ, and
    coverage percentage.
  * Microwaveprop.Propagation.RainScatterClassifier — rule-based mapper
    from (band, distance, radar stats, duct flags) to one of
    :likely_rainscatter | :rainscatter_possible | :tropo_duct |
    :troposcatter | :unknown.
  * ContactWeatherEnqueueWorker learns a :radar enrichment type and
    enqueues the CV worker on contact submission; pre-2014 contacts
    (outside IEM n0q coverage) are pinned to :unavailable.
  * `mix radar_backfill` bulk-enqueues historical contacts with
    --year / --limit / --dry-run.
  * Contact detail page renders a mechanism badge with supporting
    stats (common-volume area, max dBZ, heavy-rain pixel count,
    coverage %).
2026-04-17 15:57:59 -05:00