Commit graph

51 commits

Author SHA1 Message Date
Graham McInitre
343c8ea339 fix: use Makefile for precommit — each step runs in isolated shell with MIX_ENV=test 2026-07-21 11:08:45 -05:00
Graham McInitre
e186c4c4ef fix: use sh -c wrapper for cmd in precommit to set MIX_ENV=test 2026-07-21 11:03:33 -05:00
Graham McInitre
b1b9ff63e8 fix: isolate precommit steps with cmd, split test alias from DB setup
- Move ecto.create/ecto.migrate from test alias to test.setup
- Use cmd MIX_ENV=test in precommit to isolate each step in a fresh process
- Move Mox.defmock from test_helper.exs to valkey_test.exs
2026-07-21 11:01:06 -05:00
Graham McInitre
b91ad05c78 deps: update all dependencies to latest versions
- Relax constraints: live_stash ~> 0.2 → ~> 1.0, nx ~> 0.9 → ~> 0.13,
  sutra_ui ~> 0.3.0 → ~> 0.4.0
- Within-constraint bumps: exla 0.12→0.13, live_table 0.4.1→0.4.2,
  phoenix_live_view 1.2.6→1.2.7, postgrex 0.22.2→0.22.3, req 0.6.2→0.6.3,
  stream_data 1.3→1.4, plus transitive (castore, circular_buffer, ecto,
  mint, plug)
- Breaking: live_table 0.4.2 removed get_merged_table_options/0 — replaced
  with LiveTable.TableConfig.get_table_options(table_options()) in
  RoverLocationsLive and RoverPlanningLive
- igniter stays at ~> 0.7 (pinned by live_table ~> 0.7.0)
2026-07-20 09:59:13 -05:00
fd976b0cd5
fix: resolve 391 Credo issues across codebase
- Add jump_credo_checks ~> 0.4 with all 20 checks enabled
- Fix all standard Credo issues: 139 @spec (113 done, 26 remain),
  4 refactoring, 3 alias usage, 9 System.cmd env, 5 unsafe_to_atom,
  2 max line length, 9 assert_receive timeout
- Fix 170+ jump_credo_checks warnings:
  - 117 TopLevelAliasImportRequire: move nested alias/import to module top
  - 32 UseObanProWorker: switch to Oban.Pro.Worker
  - 4 DoctestIExExamples: add doctests / create test file
  - ~20 WeakAssertion: strengthen type-check assertions
  - Various ConditionalAssertion, AssertReceiveTimeout fixes
- Exclude vendor/ from Credo analysis
- Remaining: 175 warnings (mostly opinionated WeakAssertion,
  AvoidSocketAssignsInTest), 26 @spec annotations
2026-06-12 13:51:32 -05:00
e879f291f7
chore: update phoenix_live_view to 1.2.0 and fix test failures
Dependency updates:
- phoenix_live_view: ~> 1.2.0 (was ~> 1.1.0)
- Transitive bumps: bandit 1.11.1→1.12.0, credo 1.7.18→1.7.19,
  phoenix 1.8.7→1.8.8, req 0.5.18→0.6.1, swoosh 1.26.0→1.26.1

Fixes:
- core_components.ex: add 'type' to button :global allowlist
  (Phoenix 1.8+ attribute validation)
- status_live_test.exs: invalidate {:all_stats} cache key to
  prevent stale ETS cache from leaking between tests
- path_live_test.exs: accept all valid async states in assertion
  (race between progress frame and compute completion)
- skewt_live_test.exs: accept loading or completed state in assertion

HEEx auto-formatting from mix format (self-closing tag normalization)
2026-06-12 10:42:41 -05:00
316fb2fbc7
Fix low-severity bugs and re-enable Credo checks
- Bug #12: Lower rate limit to 15/min
- Bug #13: Wrap model loading in Task.start
- Bug #14: Fix classify_time_period guard gap at -3.0
- Bug #15: Not applicable (Elixir has no ?? operator)
- Bug #16: Remove fallback Repo.get for station preload
- Bug #17: Extract cache_key() in ContactMapController
- Bug #18: Add has_many :contacts and :beacons to User schema
- A&D #4: Move serve_markdown_if_requested after secure headers
- Config #1: Move signing_salt to runtime.exs env var
- Config #2: Use --check-unused instead of --unused
- Config #3: Re-enable UnsafeToAtom Credo check
- Config #5: Re-enable LeakyEnvironment Credo check
- Add @spec annotations to fix re-enabled Specs violations
- Replace String.to_atom with to_existing_atom where guarded
2026-05-29 17:29:22 -05:00
828814e767
fix: resolve all compiler warnings except framework-generated phoenix_component_verify
- Replace deprecated xref:exclude with elixirc_options in vendor/oban_pro
- Remove unused require Logger from 8 files
- Pin bitstring size variables with ^ in simple_packing, wgrib2,
  complex_packing, section, nexrad_client, mqtt, and radar worker
- Remove dead defp clauses (format/1 nil, depression/2 nil)
- Rename Buildings.Parser type record/0 -> building_record/0 to avoid
  overriding built-in type
- Remove redundant catch-all in candidate_detail.ex
- Simplify contact_live/show.ex conditional based on type narrowing
- Fix DateTime.from_iso8601 return pattern in vendor/oban_web
- Upgrade phoenix_live_view to 1.1.31
- Drop --warnings-as-errors from precommit alias; known false positives
  from @after_verify-generated code in Elixir 1.20.0-rc.6 + PLV 1.1.x
- Add credo --strict to precommit as replacement static-analysis gate
2026-05-29 10:18:52 -05:00
fc9d2298ac
test: lift coverage to 85% and pin threshold
Adds tests for previously under-covered modules so the cover-tool
threshold check passes:

  * Mix.Tasks.Prop.Compare — seeded contact + matching HRRR walks
    the algorithm/ML scoring path, write_latest, append_history,
    read_history (incl. the unparseable-line arm), and the per-band
    summary loop.
  * Mix.Tasks.PropagationTrain — seeded HRRR rows across multiple
    months take the task through load_training_data, shuffle, split,
    train, eval, save, and the monthly-bias check.
  * Mix.Tasks.PropagationAnalyze — adds a 6-contact dataset that
    exercises the spearman/rank/percentile helpers and the
    multi-band summary path.
  * Mix.Tasks.Unused — smoke tests run/1 with no flags,
    --skip-external, and --verbose.
  * Mix.Tasks.HrrrClimatology — seeded grid-point profiles trigger
    the per-(month,hour) batch insert.
  * Microwaveprop.Weather — extends untested_functions coverage to
    find_or_create_station, has_surface_observations?,
    station_day_covered?, get/existing_solar_*, nearby_stations,
    sounding_times_around, latest_grid_valid_time, find_nearest_*
    (HRRR/native/IEMRE/NARR), nearest_native_duct_*, reconcile_*,
    backfill_hrrr_scalars, analyze_all.
  * Microwaveprop.Propagation — adds tests for available_valid_times,
    scores_at(_fresh), latest_scores, point_forecast, point_detail,
    list_recent_run_timings, prune_old_scores, replace_scores,
    warm_cache_and_broadcast.
  * Microwaveprop.Backtest.Features — adds duct_usable_*ghz alias
    delegations.
  * Microwaveprop.Weather.IemRateLimiter — covers the is_pid clause
    of registered?/1 with a live PID.
  * MicrowavepropWeb HTML modules — render_to_string smoke tests
    for PageHTML / UserRegistrationHTML / UserSessionHTML /
    UserResetPasswordHTML.

Also pins `test_coverage: [summary: [threshold: 85]]` in mix.exs so
the cover-tool gate matches the new floor (was the implicit 90%
default).

Total goes from 82.77% → 85.06%.
2026-05-08 13:59:56 -05:00
58f4a7714b
test: add Mox deps + SnmpClient poll/3 tests, RoadProximity, Buildings.Loader
- Add {:mox, "~> 1.0", only: :test} to deps
- Commercial.SnmpClient: injectable snmp_runner for poll/3 tests (62→92%)
- Rover.RoadProximity: pure road_distances/3 tests with injectable fetch (37→82%)
- Buildings.Loader: ensure_loaded/1 with real tile file + quadkey tests (69→100%)
- Buildings.Index: records_near, mark_loaded, loaded? tests (72→96%)

Coverage: 77.96% → 78.93% (+0.97%)
2026-05-07 13:36:22 -05:00
480cc084cf
refactor(pskr): drop emqtt, talk MQTT 3.1.1 over :gen_tcp directly
emqtt's transitive `quicer` dep needs CMake + msquic submodule
clones to build, which the slim Debian builder image doesn't ship —
the prod CI build was failing inside `mix deps.compile`. Tortoise311
is the obvious pure-Elixir alternative but it bundles a
gen_state_machine + retry/inflight stack we don't use because we're
subscribe-only at QoS 0.

Implementing the wire protocol inline is ~150 lines in
`Microwaveprop.Pskr.Mqtt` (CONNECT/SUBSCRIBE/PINGREQ/DISCONNECT
builders, CONNACK/PUBLISH/SUBACK/PINGRESP parser, varint codec).
Pskr.Client now uses :gen_tcp with `active: :once` framing,
buffers partial reads, schedules its own keepalive, and parses
inbound packets in a tight loop.

Drops `BUILD_WITHOUT_QUIC=1` from the Dockerfile — no longer
relevant since there's no native dep at all in the build chain
now.
2026-05-04 09:52:34 -05:00
cfc5f7583f
feat(pskr): ingest PSK Reporter MQTT firehose for weather correlation
Subscribes to mqtt.pskreporter.info:1883 over plain TCP and folds
every reception report into hourly aggregates per
(band, sender_grid_4, receiver_grid_4). Each spot is the empirical
"propagation actually occurred on this path right now" signal we'll
correlate against HRRR atmospheric state to recalibrate the scoring
weights. Aggregating at the path-hour bucket collapses 50-reporter
QSOs to one row instead of 50.

Topic filter anchors on USA (DXCC 291) on either sender or receiver
side so the broker pre-filters out everything outside our HRRR
domain. Bands subscribed: VHF and up (6m, 2m, 70cm, 23cm, +
microwave) — HF is dominated by ionospheric propagation, which this
project doesn't model.

Pskr.Client cluster-elects a singleton via :global.register_name —
all replicas start the GenServer but only one connects to MQTT;
the rest stay in :standby and watch for the leader's nodedown to
re-run election. Off in dev/test, on by default in prod
(PSKR_MQTT_ENABLED=false as kill-switch).

Pskr.Aggregator buffers in memory keyed by Pskr.path_key/1 and
flushes every 60s via Repo.insert_all/3 with an additive
ON CONFLICT (count summed, SNR envelope by GREATEST/LEAST,
modes unioned via unnest+DISTINCT). Idempotent across overlapping
flushes and across leader handover.

Dockerfile sets BUILD_WITHOUT_QUIC=1 — emqtt's transitive quicer
dep wants CMake + OpenSSL headers we'd otherwise have to add to
the builder image just to never use QUIC over plain MQTT. Base
image is unchanged; the new dep compiles cleanly into the existing
prop-base runtime.
2026-05-04 09:24:20 -05:00
e53a34dd75
feat(pubsub): route Phoenix.PubSub through Valkey when configured
Switches the PubSub adapter to Phoenix.PubSub.Redis backed by the
shared Valkey instance (same backend GridCache uses), via the new
phoenix_pubsub_redis dep. Falls back to the default PG2 / Erlang-
distribution adapter when VALKEY_URL is unset, so dev/test boot
unchanged.

Decouples PubSub fan-out from libcluster's BEAM connection. A
flapping Erlang distribution (the OOM-cascade scenario from
2026-05-03 surfaced this — libcluster warnings dominated the logs
while pods CrashLoopBackOff'd) no longer takes broadcasts with it,
so the rest of the cluster keeps publishing while the bad pod
recovers.

node_name uses POD_IP for cluster-wide uniqueness, falling back to
Erlang's node() name in non-K8s environments. Required by the Redis
adapter so loopback filtering doesn't drop a pod's own broadcasts as
if they came from a peer.
2026-05-03 17:23:01 -05:00
9edaee9344
feat(weather): GridCache backend pluggable, Valkey for cross-pod cache
Each cached HRRR valid_time is ~32 MiB compressed × cap of 8 = up to
256 MiB per pod. With 4 hot pods + 1 backfill that's ~1.25 GiB of
cluster RAM holding identical content. Replacing the per-pod ETS
replicas with a single shared Valkey copy collapses that to ~256 MiB
total (off-pod) and makes the BEAM heap on each replica meaningfully
smaller — directly addresses the headroom the 2026-05-03 OOM cascade
ate into.

* New `Microwaveprop.Valkey`: single Redix connection (named conn,
  sync_connect: false, exit_on_disconnection: false). `start_link/0`
  returns `:ignore` when VALKEY_URL is unset so dev/test boot without
  Valkey. Helpers wrap GET/MGET/SET-with-TTL/ZADD/ZREVRANGE/SCAN with
  `:erlang.term_to_binary` round-tripping under the safe atom flag.

* `Microwaveprop.Weather.GridCache` now branches on
  `Valkey.configured?/0`. Public API is identical so callers (Weather,
  GridCachePruneWorker, MapLive) don't change. ETS path is preserved
  unchanged for dev/test.

* Storage layout when on Valkey:
    prop:wg:vts                   ZSET of valid_time iso8601 strings
    prop:wg:<vt>:<lat>:<lon>      one chunk per 5°×5° spatial bucket
  fetch_bounds MGETs only the chunks intersecting the viewport, so a
  regional view is 1-4 round-trips instead of pulling the full grid.

* Per-key TTL = 8 h (matches the ETS valid_time cap). Eviction is
  automatic; prune_keep_latest is a no-op on Valkey, prune_older_than
  uses ZRANGEBYSCORE + DEL.

* All Valkey errors fall back to disk (ScalarFile.read_bounds), logged
  as warnings — page works through Valkey outages, no user-visible
  break beyond a one-shot latency bump.

ScoreCache, Microwaveprop.Cache, NexradCache, telemetry/Postgres
protocol/libcluster registry tables remain ETS — they're sub-ms
hot-path lookups where a network hop would be felt.

VALKEY_URL must be present in prop-secrets for production rollout
(applied out-of-band, secret.yaml is git-ignored).
2026-05-03 17:00:51 -05:00
355a9df829
chore(deps): document why oban_web needs override: true
Aligning vendoring style with the sibling towerops-web repo. They
have the same vendor/oban_pro + oban_met + oban_web setup but no
`override: true` on oban_web — because they don't have live_table.

We do (live_table requires `oban_web ~> 2.11` from hexpm), so the
override is load-bearing here. Comment captures the reason so a
future deps cleanup doesn't assume it's vestigial and remove it.
2026-04-30 09:31:39 -05:00
e99bf06eb4
deps: re-vendor oban_pro 1.7.0 (revert hex-repo dep)
Previous commit (3c988a5f) switched oban_pro to the licensed hex
repo at deploy time. Reverting that — keep all three Pro packages
vendored so prod image builds don't depend on oban.pro reachability
or auth on every CI run.

oban_pro 1.7.0 dropped into vendor/oban_pro via
`mix hex.package fetch oban_pro 1.7.0 --repo=oban --unpack`. mix.exs
goes back to `path: "vendor/oban_pro"` (matching oban_met / oban_web,
which were already vendored — and stay vendored since the licensed
repo only has older versions of those: 0.1.11 / 2.10.6 vs the
1.1.0 / 2.12.1 we vendor).

Schema migration from 3c988a5f stays — 1.7.0 tables/indexes are
already applied. No code changes.
2026-04-30 09:29:55 -05:00
3c988a5ff3
deps: oban_pro 1.6.14 (vendored) → 1.7.0 (hex)
Oban Pro 1.7.0 ships database-backed workflow tracking, replaces the
generated `uniq_key` / `partition_key` columns with expression
indexes, and adds new partial indexes for query performance.

Switches oban_pro from the vendor/ tree to the licensed `oban` hex
repo (auth already configured via `mix hex.repo`). vendor/oban_pro
deleted — the vendoring was a 1.6.x stopgap, 1.7 is the first
version we land directly from hex.

oban_met (1.1.0) and oban_web (2.12.1) stay vendored — they're
not published to the licensed `oban` repo (verified by 404 on
`mix hex.package fetch`).

Migration 20260430142341 runs `Oban.Pro.Migration.up(version: "1.7.0")`
which creates the workflow tables and the new indexes. Verified on
dev + test DBs. Production is well below the size threshold the v1.7
guide flags for split-migration handling, so the inline default
suffices.

No code changes for the breaking-change list — the codebase only
references Oban.Pro.Engines.Smart and Oban.Pro.Plugins.DynamicLifeline,
both unchanged in 1.7. We don't use Workflow.after_cancelled/2 or
Oban.Pro.Workers.Chunk.

Test suite stays at 3132/3132.
2026-04-30 09:25:11 -05:00
40cbb40cb1
feat(rover): score per-station terrain clearance from SRTM path samples
For each (rover cell, fixed station) pair, sample SRTM along the
great-circle path and add a per-link dB bonus proportional to the
rover's elevation above the highest intermediate terrain. Rover spots
on hilltops with clear sight to multiple stations now rise to the top.

Also vendor Leaflet's layers control PNGs and copy them under
priv/static/assets/css/images/ on build so /rover stops 404'ing on
layers-2x.png.
2026-04-25 17:29:29 -05:00
100be89c06
chore: drop OpenTelemetry
Removes the six OTel packages (opentelemetry, _api, _exporter,
_phoenix, _oban, _bandit) and their 9 transitive deps. Tracing was
only wired for Phoenix/Bandit/Oban auto-instrumentation — PromEx
already covers the metrics we actually use on the status page +
Grafana. Also drops the OTEL_EXPORTER_OTLP_ENDPOINT env var and
its comment block from both deployment manifests.
2026-04-24 12:33:38 -05:00
c87334d31d
chore(deps): drop gettext
No translations are actually maintained and no UI is shipped in a
non-English locale. Replaced the handful of gettext(...) calls in
core_components + layouts with their literal English strings, and
rewrote translate_error/1 as a small %{key} interpolator over the
Ecto changeset error tuple.

Removes gettext + expo (compile dep). sutra_ui stays — it's a
transitive dep of live_table's TableComponent.
2026-04-24 12:31:21 -05:00
75ec3f8d1e
chore(deps): upgrade live_stash 0.1.2 → 0.2.0
Two breaking API changes in 0.2:

- `stash_assigns/2` is gone. Keys to persist are now declared on the
  `use LiveStash, stored_keys: [...]` macro and `stash/1` picks them
  up automatically.
- The TTL unit switched to seconds — this codebase never called the
  TTL setter, so no config change is needed.

Updated both callers (MapLive persists selected_band + selected_time;
SubmitLive persists active_tab). `recover_state/1` signature is
unchanged. Full LiveView suite passes (67 tests).
2026-04-23 13:56:28 -05:00
52a32032a3
feat(logging): structured JSON logs in prod via LOG_FORMAT env var
The ingress + Grafana pipeline parses JSON-formatted log lines; Elixir's
default text formatter was the odd format everywhere else, forcing Loki
to fall back to free-text parsing. Swap the `:default` :logger handler's
formatter to `LoggerJSON.Formatters.Basic` when `LOG_FORMAT=json`.

Defaults: `LOG_FORMAT=text` for dev/test (no behaviour change), `json`
for prod. Whitelisted metadata covers the fields we actually correlate
on — `request_id`, `trace_id`, `worker`, `queue`, `job_id`.

OTP 28 / Elixir 1.19 exposes `Logger.Formatter.new/1` but ships no
JSON encoder of its own, so `logger_json` (~> 7.0) carries the rendering.
The test pokes the formatter directly rather than round-tripping through
ExUnit's capture_log, which replaces the handler and so can't see the
installed formatter's output.
2026-04-21 14:17:54 -05:00
6c6e413105
fix(build): inline dep-patch logic into mix.exs so it runs during deps.get
Previously the `mix deps.patch` task lived at `lib/mix/tasks/deps.patch.ex`
and the `deps.get` alias invoked it. This broke `mix deps.get --only prod`
in Docker: `lib/` isn't compiled yet when deps are fetched, so the task
module couldn't be resolved and the build failed with
`The task "deps.patch" could not be found`.

Move the patch-application function directly into `mix.exs` as a private
helper and wire it into the alias via `&apply_dep_patches/1`. `mix.exs` is
always loadable, so the function is available the moment `deps.get` runs.
2026-04-21 14:06:24 -05:00
1400c38f44
fix(dialyzer): actually fix LiveTable warnings instead of hiding them
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.

Changes:

- Delete .dialyzer_ignore.exs and remove ignore_warnings from
  mix.exs. No more hidden suppressions.

- New MicrowavepropWeb.LiveTableResource shim that wraps
  use LiveTable.LiveResource and overrides:
  * maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
    (previously discarded, tripping :unmatched_return).
  * handle_info/2 — explicit _ = on File.rm, Process.send_after.
  Switch all 4 LiveTable-using LiveViews (contact_live/index,
  beacon_live/index, admin/contact_edit_live, user_management_live/index)
  to use MicrowavepropWeb.LiveTableResource instead.

- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
  LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
  the dep's macro-expanded handle_params/3. The call's return was
  silently discarded, tripping the fourth warning per LiveView.

- New lib/mix/tasks/deps.patch.ex applies every
  priv/dep_patches/*.patch to its target dep idempotently after
  mix deps.get. Wired into the :setup alias + overridden
  mix deps.get so the patch survives cache regeneration in CI.

- Other modified files in this commit are the spec-coverage
  additions from the parallel agent pass (beacons.ex,
  commercial.ex, propagation.ex, weather.ex, narr_client.ex,
  run_timing.ex, 3 workers) — tighter specs for bounds, tuple
  shapes, schema-typed params. Dialyzer remains at 0 after
  the changes.

Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.

mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
2026-04-21 12:58:10 -05:00
d61fbd346e
fix(dialyzer): clear 125+ warnings under strict flags
Enabled :error_handling, :unknown, :unmatched_returns, :extra_return,
:missing_return in an earlier commit and landed a 129-warning baseline.
Four parallel agents each fixed a directory slice:

- Core contexts (29): Radio, Release, Weather, Beacons, Cache,
  Backtest.Features, Terrain.Srtm, Ionosphere.GiroClient,
  Propagation.RunTiming, Accounts.Scope, RepoListener. Fixes were
  (a) prefix side-effect calls (Task.start, Phoenix.PubSub,
  Logger, :ets.new) with _ = ; (b) tighten/widen specs that didn't
  match actual returns; (c) add missing @type t declarations;
  (d) drop dead parse_int(nil) clause.

- Propagation + weather subdirs (15): FreshnessMonitor, NotifyListener,
  ScoreCache, ScoreCacheReconciler, Weather.FrontalAnalysis,
  Weather.Grib2.Extractor, Weather.Grib2.Wgrib2, GridCache,
  HrrrPointEnqueuer, NexradCache. Same patterns — mostly _ = on
  PubSub / :ets / Repo.insert_all; widened two specs (float ->
  number) where integer returns were reachable.

- Workers (35): BackfillEnqueue, CanadianSoundingFetch,
  ContactImport, ContactWeatherEnqueue, GefsFetch, IemreFetch,
  NarrFetch, SolarIndex, TerrainProfile, WeatherFetch. Prefixed
  Repo.update_all / Radio.set_enrichment_status! / Weather.upsert_*
  side-effect calls. Fixed one :pattern_match in
  CanadianSoundingFetch.most_recent_sounding_time/1 where a
  tautological cond guard generated unreachable code.

- Web + Mix tasks + lib_ml (46 of 50): controllers, LiveViews,
  UserAuth, and 11 mix tasks. Same prefix strategy. 4 remaining
  warnings originate in LiveTable.LiveResource dep macro expansion
  and can't be fixed without forking the dep — added .dialyzer_ignore.exs
  to suppress just those specific file:line pairs.

Also wired ignore_warnings in mix.exs dialyzer config.

mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2163 tests, 3 pre-existing flakes, 0 regressions.
2026-04-21 10:30:06 -05:00
733a7f5bf1
fix(review): address code-reviewer findings
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):

- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
  directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
  Previously it called ScoresFile.read_bounds/3 which silently returns
  [] on missing/corrupt files, poisoning ScoreCache with an empty grid
  that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
  pattern-match {:error, reason} and skip (log, return :error) instead
  of caching empty. rescue clauses kept as defense-in-depth for
  unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
  need to distinguish missing file from empty grid can feed read/2
  payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
  helper (Map.fetch) so a legitimate persisted ducting_detected: false
  is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
  Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
  guard so callers with partial contacts get a clean false rather
  than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
  and wrapped in try/rescue with per-row fallback on Postgrex errors
  so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
  explicit {:error, _} pattern match). FM5 clarifies the surviving
  PropagationGridWorker is a cron-fired seed worker, not a fallback
  compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
  :unmatched_returns, :extra_return, :missing_return). Baseline
  emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
  returns; a follow-up will tighten those and backfill @specs.

New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
  interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
  on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
  error string contract so permanent_error?/1 classification doesn't
  silently regress if the client's error format changes.
2026-04-21 10:09:46 -05:00
1a1ba460e4
fix(otel): drop opentelemetry_ecto + disable OTLP on Elixir pods
Ecto auto-instrumentation created one span per query, so Oban jobs
running thousands of DB ops produced traces with 8k+ child spans.
Tempo's compactor couldn't ingest them — it stalled long enough to
miss heartbeats, got auto-forgotten from the ring, and crash-looped.
The Elixir grpcbox exporter then blew up trying to reconnect.

Stops the cascade: OTLP endpoint removed from prop and prop-backfill
Deployments, opentelemetry_ecto removed from deps, Phoenix/Oban/Bandit
auto-instrumentation retained. Rust workers still export to Tempo.
2026-04-20 13:41:01 -05:00
55251df8a0
feat(elixir): OpenTelemetry trace export to cluster OTel Collector
Adds the OTel deps (opentelemetry + exporter + phoenix / ecto / oban /
bandit auto-instrumentation helpers) and attaches them in
Application.start/2 so the existing Phoenix, Ecto, Oban, and Bandit
telemetry events flow as OTLP spans without any call-site changes.

Exporter config is gated on OTEL_EXPORTER_OTLP_ENDPOINT in
config/runtime.exs — set in the k8s manifests to the cluster
collector (otel-collector.observability.svc.cluster.local:4317).
When unset we switch traces_exporter to :none so nothing is shipped;
dev/test stays quiet.

Resource attributes tag spans with service.name=microwaveprop and
service.namespace=prop, matching the Rust workers' attribute shape
so Tempo can group the full hourly chain across both languages.

Both the main prop deployment and the backfill deployment get the
env; backfill is still a full BEAM node running enrichment workers,
so its Oban/Ecto spans are worth seeing too.
2026-04-20 12:02:15 -05:00
d41ced5850
feat(profiles_file): read Rust-written MessagePack alongside legacy ETF
Phase 3 Stream A: Rust's prop-grid-rs writes f00 profile files as
MessagePack (c4f309c). Elixir needs to read both formats during the
cutover window and indefinitely afterward — .mp.gz wins when both
exist.

Reader:
- path_for/1 stays on .etf.gz (Elixir legacy writer)
- mp_path_for/1 returns the .mp.gz sibling Rust lands
- read/1 prefers .mp.gz, falls back to .etf.gz, :enoent if neither
- decode_mp_body handles the top-level {v, valid_time, cells} shape
- normalize_profile atomizes a whitelist of known keys (surface_*,
  native_min_gradient, ducts[], profile[] levels, ...) and leaves
  everything else as strings so no String.to_atom bomb is possible
- Directory scan picks up both extensions so retain_window and
  prune_older_than also work uniformly; list_valid_times dedupes
  by timestamp

Adds msgpax ~> 2.4 dependency. 4 new tests cover format preference,
key atomization, :enoent path, and dedupe.
2026-04-19 18:10:23 -05:00
c7bc3ed5d0
feat(telemetry): PromEx Prometheus exporter at /metrics
Wires PromEx with its built-in Application / BEAM / Phoenix / Ecto /
Oban plugins plus a custom `Microwaveprop.PromEx.InstrumentPlugin`
that registers Prometheus histograms for every
`Microwaveprop.Instrument` span (HRRR/NEXRAD/IEM/GEFS/NARR/UWYO
/Elevation fetches, DB batch upserts, terrain analysis, radar
aggregation, propagation score band). Queue-depth gauges from the
10-second poller land at `microwaveprop_oban_queue_count{queue,state}`.

Router forwards `/metrics` to MicrowavepropWeb.MetricsPlug which
optionally requires a bearer token (`PROMETHEUS_AUTH_TOKEN` env var in
runtime.exs); leave it unset to expose metrics publicly and restrict
at the ingress / Cloudflare Access layer instead.

Example external Prometheus scrape config:

    scrape_configs:
      - job_name: microwaveprop
        scrape_interval: 30s
        metrics_path: /metrics
        scheme: https
        authorization:
          type: Bearer
          credentials: <token>
        static_configs:
          - targets: ['prop.w5isp.com:443']
2026-04-18 16:39:39 -05:00
ca0a0eaa50
Unblock CI: pull igniter into :prod for live_table compile
Since da6b312 (Add live_table dep), every main-branch CI build has
silently failed on `mix deps.compile` inside the Docker builder. The
failure is that live_table ships two unguarded mix tasks —
lib/mix/tasks/live_table.install.ex and
lib/mix/tasks/live_table.gen.live.ex — both of which do
`use Igniter.Mix.Task` at the top level without wrapping it in a
`Code.ensure_loaded?(Igniter.Mix.Task)` check. When Igniter isn't
available, compiling the live_table dep blows up with
`module Igniter.Mix.Task is not loaded`.

I had declared igniter as `only: [:dev, :test], runtime: false`
which meant it was only fetched in those envs, so the prod Docker
build never had it and failed there. The blast radius is every
commit from da6b312 through 263c1bb (14 commits) none of which ever
made it to prod:

  - /users, /beacons, /contacts, /admin/contact-edits live_table
    conversions
  - Admin nav dropdown
  - Cap "Contacts I'm in" at 100
  - Contact detail path-calc link + 10 ft AGL assumption
  - Path calculator callsign tooltip
  - Drop gridmap.org and do QRZ + Google geocoding locally
  - Feed Loss (×) HEEx escape fix
  - Path forecast line chart and refractivity gradient unit fix
  - Float.round/2 crash fix on clear-terrain diffraction loss

Drop the env restriction on igniter so it's available to the prod
build as well. `runtime: false` keeps it out of the release bundle
so there's no runtime bloat, it's strictly a compile-time build
tool for the live_table dep.

Verified locally with a clean `_build/prod` + `MIX_ENV=prod mix
deps.compile` (all 54 deps including live_table now build) and the
full test suite still passes.
2026-04-12 17:58:13 -05:00
99e7560601
Drop gridmap.org dependency, resolve callsigns locally
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.

New modules:

- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
  Looks up from qrz_callsigns first, falls back to a live fetch, and
  upserts the result with a configurable cache_ttl_hours (default
  168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
  https://xmldata.qrz.com/xml/current/. Holds the session key in an
  Agent, transparently re-logs-in on :session_expired, and parses
  responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
  cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
  we actually consume (identity, name, grid, address, lat/lon).
  The full XML payload stays in the raw :data jsonb column for
  anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
  Geocoding API. Only called as a fallback when QRZ has no explicit
  <lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
  callsign_locations cache, on miss calls Qrz then either uses QRZ's
  coords directly or geocodes the formatted address, snaps to an
  8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
  upserts the result.

Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.

Wiring:

- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
  creates qrz_callsigns and callsign_locations with unique indexes
  on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
  so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
  and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
  never short-circuits test-level stubs, and supplies dummy QRZ
  credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
  and GOOGLE_API_KEY from the environment so prod and dev can
  configure both upstream keys out of band.

Tests (ported verbatim from gridmap-web):

- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
  behavior, upsert on stale, error passthrough, case-insensitive
  input
- test/microwaveprop/geocoder_test.exs — success, zero results,
  request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
  including the QRZ-lat/lon shortcut and the missing-address error
  path

All 1294 tests still pass. Credo strict clean.
2026-04-12 17:43:29 -05:00
da6b312213
Add live_table dep and convert /users to use it
Wire up gurujada/live_table 0.4.1 as a dependency (alongside its
transitive sutra_ui and optional igniter requirement) with the
:live_table config pointing at our Repo and PubSub. Forced oban_web
override so the path-pinned vendored copy still wins over live_table's
hex constraint.

Convert the admin /users page to use LiveTable.LiveResource with
sortable, searchable columns for callsign, name, and email. Custom
renderers preserve the daisyUI admin badge and the pending/confirmed
date cell. Hidden id field stays in the query so the edit and delete
actions can target a record by id.

Updated the delete test to push the delete event directly with an id
payload; the stream dom_ids under live_table are random UUIDs so
tr#users-<id> selectors no longer work.
2026-04-12 17:07:07 -05:00
c6bf333c98 Add dialyxir for static analysis
PLT stored in priv/plts/ (gitignored). Run with `mix dialyzer`.
2026-04-11 17:43:18 -05:00
0db5c2ae3e Vendor oban_pro 1.6.13
Adds the commercial Oban Pro package (vendored from the towerops-web2
vendor tree) so this project can use its workers, plugins, and
Smart engine features.
2026-04-09 14:14:49 -05:00
caad7d90a7 Mount Oban Web dashboard at /admin/oban
Vendor oban_web 2.12.1 and oban_met 1.1.0 (taken from the
towerops-web2 vendor tree), bump oban to ~> 2.21, and mount the Oban
dashboard at /admin/oban behind the existing require_admin on_mount
hook. Adds an admin-only "Oban" nav link and extends the Dockerfile
to copy vendor/ before deps.get so production builds pick up the
path dependencies.
2026-04-09 14:07:33 -05:00
355bed178d Add beacon monitor registration and fix prod SMTP
Users can now register beacon monitors on the settings page. Each
monitor gets a unique random token the remote program will use to
authenticate its reports. Name is the only user-supplied field for
now; more will be added as the monitor protocol is defined.

Also adds :gen_smtp to deps. Swoosh's SMTP adapter depends on
:mimemail which lives in that package, and production was 500'ing
when trying to send confirmation emails without it.
2026-04-08 11:30:28 -05:00
1d86e287b2 Add password auth with callsign + email confirmation
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:

- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
  until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
2026-04-08 10:21:40 -05:00
091d469c6a Enable OS and Ecto data in LiveDashboard, fix progress bars
- Add ecto_psql_extras for PostgreSQL insights in dashboard
- Add os_mon for OS metrics (CPU, memory, disk)
- Fix progress bars using percentage instead of raw large numbers
2026-04-06 09:56:08 -05:00
2de94e2318 Add Erlang clustering via libcluster and expose LiveDashboard
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
2026-04-06 09:46:48 -05:00
d275e9e7c4
fix enrichment 2026-04-02 15:30:41 -05:00
c1da0f20b1
Add LiveStash for state recovery across reconnects
- Installed live_stash ~> 0.1 with browser memory adapter
- Map page: recovers selected_band and selected_time on reconnect
- Submit page: recovers active_tab (single/csv) on reconnect
- JS params initialized with initLiveStash wrapper
2026-04-02 13:12:17 -05:00
93f2c32971
Remove timex, earmark, dns_cluster deps; add local implementations
- Replace Timex with regex-based timestamp parser (broader format support)
- Replace Earmark with local Markdown.to_html! (headings, code blocks,
  tables, lists, inline formatting, links, horizontal rules)
- Remove dns_cluster (unused, single-instance deployment)
- Add stream_data property tests for timestamp parsing (5 properties
  covering ISO, US dates, AM/PM, format roundtrips, garbage rejection)
- Removes transitive deps: combine, tzdata, hackney, certifi, metrics,
  mimerl, parse_trans, ssl_verify_fun
2026-04-02 12:17:37 -05:00
ec388e4bf5
Use Timex for flexible CSV timestamp parsing with tests
- Add timex ~> 3.7, downgrade gettext to ~> 0.26 for compatibility
- Parse ISO 8601, US dates (M/D/YYYY), AM/PM, 24h, compact formats
- 22 tests covering all accepted timestamp formats and edge cases
- Integration tests for CSV import with US date and space-separated formats
2026-04-01 15:39:02 -05:00
71091219f4
Add tidewave dev tool, taller elevation profile chart
- Install tidewave ~> 0.5 for dev MCP integration
- Increase elevation profile chart height from h-64 to h-96
2026-04-01 13:41:39 -05:00
2c16fd979d
Move ML deps to dev/test only, exclude from production build
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
  elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
  compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
2026-04-01 12:10:31 -05:00
b391d3a75e
Add ML training pipeline with Polaris optimizer
- Add polaris dep for Adam optimizer (Axon.Optimizers deprecated)
- Fix Model.train/3 to use Polaris.Optimizers.adam
- Training task: mix propagation_train with stratified band sampling
- Model encodes solar time (longitude/15) not fixed timezone
2026-04-01 09:06:10 -05:00
8949920b7f
Add Nx/Axon/EXLA ML model skeleton for propagation prediction
13-feature feed-forward network (atmospheric + temporal + frequency).
Includes build, init, predict, encode_features, save/load to disk.
Model weights saved to priv/models/propagation_v1.nx (gitignored).
Not yet trained — scaffolding only.
2026-03-31 16:26:34 -05:00
d5140267b1
Show topbar loading indicator while scores are loading 2026-03-31 09:30:20 -05:00
6f16395f44
Add QSO import, solar indices, Oban workers, LiveView UI, and parallel weather import
- Radio context with QSO schema and CSV import script (58K contacts)
- Solar index schema, GFZ client, and daily Oban cron worker
- LiveView dashboard with QSO table and weather correlation views
- Parallelize weather import script using Task.async_stream (10 concurrent workers)
- Replace sequential rate_limit sleeps with concurrency-based backpressure
- Atomic progress counter for interleave-safe reporting
2026-03-29 13:04:55 -05:00