Commit graph

1895 commits

Author SHA1 Message Date
Graham McInitre
cdf4c75dd5 feat: clickable band filters on PSK Reporter tab, fix sync blocking in refresh
- Fix handle_info(:refresh_spots) to use start_async instead of blocking
  LiveView process on DB fetches
- Add require Logger to fix pre-existing compiler warning
- Make band counts clickable: clicking a band filters table to that band's
  last 100 spots; clicking again clears the filter
- Highlight active filter badge, show Clear filter link when filtered
- Update subtitle and empty state dynamically based on filter
- Filter persists across auto-refresh (60s)
- Add 5 test cases for band filtering behavior
2026-07-20 12:41:18 -05:00
Graham McInitre
b91ad05c78 deps: update all dependencies to latest versions
- Relax constraints: live_stash ~> 0.2 → ~> 1.0, nx ~> 0.9 → ~> 0.13,
  sutra_ui ~> 0.3.0 → ~> 0.4.0
- Within-constraint bumps: exla 0.12→0.13, live_table 0.4.1→0.4.2,
  phoenix_live_view 1.2.6→1.2.7, postgrex 0.22.2→0.22.3, req 0.6.2→0.6.3,
  stream_data 1.3→1.4, plus transitive (castore, circular_buffer, ecto,
  mint, plug)
- Breaking: live_table 0.4.2 removed get_merged_table_options/0 — replaced
  with LiveTable.TableConfig.get_table_options(table_options()) in
  RoverLocationsLive and RoverPlanningLive
- igniter stays at ~> 0.7 (pinned by live_table ~> 0.7.0)
2026-07-20 09:59:13 -05:00
Graham McInitre
4ac606e682 fix: ensure all unused historical NFS data gets cleaned up
Close 6 cleanup gaps on the shared /data NFS mount:

- HRDPS .hrdps.prop score files: parse_valid_time regex now matches
  the .hrdps.prop extension, so these files are found and pruned
- HRDPS scalar dirs (weather_scalars/{iso}.hrdps/): strip .hrdps suffix
  before DateTime.from_iso8601 in list_valid_time_dirs
- ScalarFile: prune_older_than was defined but never called from
  Propagation.prune_old_scores - now called alongside Scores/Profiles
- Orphan .tmp.* files: sweep all three base dirs (scores/profiles/scalars)
  for .tmp.* files left by crashed atomic-write processes
- Building cache (/data/buildings/): add MsFootprints.prune_older_than
  with 30-day mtime cutoff
- Canopy cache (/data/canopy/ + staging/): add Canopy.prune_older_than
  with 30-day cutoff, cleanup empty staging dir

All cleanup is cutoff-based (older than X time) so if the prune worker
is missed for any period it catches up fully on the next run.

PropagationPruneWorker updated to call all new cleanup functions.
2026-07-20 09:27:36 -05:00
Graham McInitre
b9aac1b210 update for ip changes 2026-07-20 09:01:36 -05:00
Graham McInitre
2e720875c4 chore: bump nix postgres from 16 to 17 2026-07-16 09:23:39 -05:00
Graham McInitre
57bd89166f chore: drop 6m from PSK Reporter subscription — only 2m and up 2026-07-16 08:40:02 -05:00
Graham McInitre
e6018c6fd9 fix: pskreporter — eliminate @spots crash, duplicate stream IDs, and slow aggregate blocking with start_async 2026-07-16 08:36:00 -05:00
Graham McInitre
5eeac23e83 refactor: extract ProfileLookup module from weather.ex (2072→1696 lines, -376) 2026-07-16 07:47:47 -05:00
Graham McInitre
695bbf5cee perf: add batch hrrr_profiles_for_contacts/1 to collapse N+1 per-contact queries into one DB round-trip 2026-07-16 07:37:34 -05:00
Graham McInitre
82e02ae0b8 perf: reuse cached hrrr_path in terrain_ready handler instead of re-fetching 2026-07-16 07:34:15 -05:00
Graham McInitre
fc51f8afe8 perf: concurrent ASOS fetches + bulk upsert in commercial poll worker
- Replace sequential Enum.each with Task.async_stream for station I/O
- Replace per-row upsert_surface_observation with batch upsert_surface_observations
2026-07-16 07:33:03 -05:00
Graham McInitre
93b8f881e2 perf: batch fixes for site-wide audit — indexes, N+1, streams, atomicity
- Add index on weather_stations(station_type, lat, lon) for nearby_stations
- Add UPPER(station1/2) expression indexes for callsign search
- Batch sync_network ~3K individual Repo.insert calls into one insert_all
- Batch Oban.insert calls in insert_unique into Oban.insert_all
- Log warnings on station elevation Repo.update errors instead of discarding
- Wrap reconcile_mission_paths delete+insert in Repo.transaction
- BeaconLive.Index: targeted stream_insert/delete instead of full re-query+push_patch
- RoverPlanningLive.Show: re-fetch single path instead of re-querying all
- PskrSpotsLive: convert to LiveView streams, add 60s auto-refresh
- ImportConfetti: add missing phx-update=ignore
2026-07-16 07:31:19 -05:00
Graham McInitre
15c4ba11a1 fix: Repo.transact -> Repo.transaction (crash on email change + token revocation) 2026-07-16 07:26:49 -05:00
Graham McInitre
c4cd659321 perf: add index on pskr_spots_hourly.last_spot_at for /pskreporter ORDER BY 2026-07-16 07:20:03 -05:00
Graham McInitre
beed193605 fix: handle GIRO API 404 cleanly — don't dump raw HTML into logs 2026-07-16 07:17:52 -05:00
Graham McInitre
65df3f9242 fix: override Repo.all/2 to accept :limit/:offset opts for Oban Pro compat 2026-07-15 14:41:02 -05:00
Graham McInitre
3297147c31 feat: add total spot count and per-band breakdown to /pskreporter header 2026-07-15 14:32:55 -05:00
Graham McInitre
51dda456aa feat: track sender/receiver callsigns in pskr spots + full-width /pskreporter page 2026-07-15 14:29:23 -05:00
Graham McInitre
a2e2776e7c fix: batch pskr calibration upsert into 1,000-row chunks to avoid Postgres 65,535 param limit 2026-07-15 14:18:43 -05:00
Graham McInitre
38ee285833 fix: add limit assign so @limit works in pskr_spots template 2026-07-15 14:15:51 -05:00
Graham McInitre
5698d32865 fix: revert Docker builder to Elixir 1.20.1 / OTP 29.0.2 2026-07-15 13:58:06 -05:00
Graham McInitre
986efb2345 Revert "fix: bump Docker debian date suffix from 20260518 to 20260713"
This reverts commit ee2d94ce57.
2026-07-15 13:57:04 -05:00
Graham McInitre
ee2d94ce57 fix: bump Docker debian date suffix from 20260518 to 20260713 2026-07-15 13:36:34 -05:00
Graham McInitre
d76c36d25f feat: add /pskreporter page showing last 100 PSK Reporter spots 2026-07-15 13:30:34 -05:00
Graham McInitre
a8ecee32b0 feat: add Nix flake dev environment with Erlang 29.0.3 + Elixir 1.20.2
- Add flake.nix + flake.lock pinning nixpkgs-unstable
- Add nix/shell.nix with PostgreSQL, Rust, Elixir LSP, pre-commit hooks
- Add shell.nix flake-compat shim for non-flake nix-shell users
- Bump Dockerfile to Elixir 1.20.2 / OTP 29.0.3
- Add nix.md with setup and troubleshooting docs
- Gitignore Nix runtime state and generated pre-commit config

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-15 12:57:10 -05:00
aa6c885683
fix: repair 18 failing tests (atom naming + stale assertions)
- Replace Module.safe_concat/1 with PID-based names in IemRateLimiterTest
  and AggregatorTest (8+6 tests) to avoid binary_to_existing_atom errors
  after Elixir 1.20 upgrade
- Fix ContactLive ShowCoverageTest assertions (3 tests) that referenced
  removed template text ('in queue') and incompatible weather_status
- Use conn.remote_ip instead of init_test_session for internal_network?
  tests since the store_remote_ip plug uses atom keys
2026-07-08 11:10:29 -05:00
ecb425780e
chore: bump deps (phoenix 1.8.9, live_view 1.2.6, swoosh 1.26.3, redix 1.6.0, websock_adapter 0.6.0) 2026-07-08 10:51:45 -05:00
5f154a20f3
refactor: apply functional programming patterns to accounts + map_live
- accounts.ex: replace if with with-chain in get_user_by_email_and_password
  for explicit pattern matching on User struct + valid_password? result.
  Collapse nested case into with in create_api_token.

- map_live.ex: extract connected? PubSub subscription block from mount into
  named subscribe_if_connected/1 function (cleaner than inline if with _ =).
  Deduplicate identical select_time + set_selected_time handlers into shared
  handle_set_time/2 helper. Extract toggle_grid + toggle_radar into shared
  handle_toggle/3 helper (functions-as-values pattern for assign key).
2026-07-01 17:56:06 -05:00
c2efead65c
chore: bump deps, fix doctests, fix credo issues 2026-07-01 17:46:03 -05:00
00f6d8248b
update elixir 2026-07-01 15:46:51 -05:00
0d00d1777c
fix: 9 bugs across tests and production code
- Fix Module.safe_concat -> Module.concat in tests with dynamic process names
  (safe_concat calls binary_to_existing_atom, but test names are newly generated)
- Fix PSKR AggregatorTest sandbox ownership by switching to async: false
- Fix MapLiveTest assert_patch: regex unsupported in LiveView 1.2, use string match
- Fix WeatherMapLiveTest toggle_grid: assert on checked attribute, not data-grid
- Fix BeaconLive.Index leaking unapproved beacons: wire data_provider to approved_beacons_query
- Fix ContactLive.Index leaking private contacts: wire data_provider with visibility filter
- Fix RecalibratorTest: train() expects factor vectors, not {vector, datetime, band} tuples
- Fix toggle_sort: compare current_field to new_field, not current_order
- Fix internal_network?: handle both atom and string session keys from Plug sessions

Test results: 3979/4001 -> 3997/4001 (18 previously-failing tests now pass)
2026-06-21 12:13:58 -05:00
6bd4361ed1
updates 2026-06-16 12:38:08 -05:00
49108423d6
cleanup 2026-06-13 14:14:18 -05:00
acfdca35df
update credo 2026-06-12 16:14:14 -05:00
cb8445f329
fix: resolve 158/183 credo --strict warnings
- Add 17 missing @spec annotations (layouts, error_json, error_html, skewt_svg)
- Move 12+ nested alias/import/require to module top level
- Add phx-change/id attributes to 11 raw HTML <form> tags
- Remove 4 unused LiveView assigns (:bounds, :data_provider)
- Add 3 missing doctest references (HrrrNativeClient, BulkFetch, Accounts)
- Break 2 long lines (path_compute.ex:382)
- Strengthen weak test assertions (is_binary→byte_size, is_list→!=[])
- Replace Module.concat with Module.safe_concat (2 occurrences)
- Replace length/1 > 0 with list != [] (9 occurrences)
- Remove no-op assert true, fix no-assertion tests

Remaining: 24 socket.assigns introspection warnings (deliberate test
pattern for observable behavior testing), 1 formatter-resistant long
line, 3 app-code usage warnings.
2026-06-12 15:47:15 -05:00
4a2f259f49
fix: additional @spec and test assertion fixes from agents 2026-06-12 13:53:25 -05:00
fd976b0cd5
fix: resolve 391 Credo issues across codebase
- Add jump_credo_checks ~> 0.4 with all 20 checks enabled
- Fix all standard Credo issues: 139 @spec (113 done, 26 remain),
  4 refactoring, 3 alias usage, 9 System.cmd env, 5 unsafe_to_atom,
  2 max line length, 9 assert_receive timeout
- Fix 170+ jump_credo_checks warnings:
  - 117 TopLevelAliasImportRequire: move nested alias/import to module top
  - 32 UseObanProWorker: switch to Oban.Pro.Worker
  - 4 DoctestIExExamples: add doctests / create test file
  - ~20 WeakAssertion: strengthen type-check assertions
  - Various ConditionalAssertion, AssertReceiveTimeout fixes
- Exclude vendor/ from Credo analysis
- Remaining: 175 warnings (mostly opinionated WeakAssertion,
  AvoidSocketAssignsInTest), 26 @spec annotations
2026-06-12 13:51:32 -05:00
e879f291f7
chore: update phoenix_live_view to 1.2.0 and fix test failures
Dependency updates:
- phoenix_live_view: ~> 1.2.0 (was ~> 1.1.0)
- Transitive bumps: bandit 1.11.1→1.12.0, credo 1.7.18→1.7.19,
  phoenix 1.8.7→1.8.8, req 0.5.18→0.6.1, swoosh 1.26.0→1.26.1

Fixes:
- core_components.ex: add 'type' to button :global allowlist
  (Phoenix 1.8+ attribute validation)
- status_live_test.exs: invalidate {:all_stats} cache key to
  prevent stale ETS cache from leaking between tests
- path_live_test.exs: accept all valid async states in assertion
  (race between progress frame and compute completion)
- skewt_live_test.exs: accept loading or completed state in assertion

HEEx auto-formatting from mix format (self-closing tag normalization)
2026-06-12 10:42:41 -05:00
5c62243719
chore: bump builder image to Elixir 1.20.1 / Erlang 29.0.2 2026-06-10 17:36:06 -05:00
9db2cd20f5
fix: resolve 7 pre-existing test failures and ScoreCache DateTime bug
- CacheTest: fix sweep timing race by using negative TTL (-1)
  instead of positive TTL (1) for already-expired entries
- ScoreCache: replace ETS match-spec DateTime comparisons with
  :ets.foldl + DateTime.compare — DateTime structs are maps in
  Elixir >= 1.15 and ETS can't compare maps with :< / :> guards
- Accounts: drop unsupported returning: true on delete_all,
  return [] for expired tokens list
- Backtest: catch ArgumentError from String.to_existing_atom
  for unknown feature names, preserving the helpful Mix.Error
- ContactLive IndexTest: invalidate monthly_bars cache before
  assertion so test data is visible
- RoverLocationsLive MapTest: invalidate cached points before
  assertion
- StatusLiveTest: add DB cleanup in setup to reduce test
  interference; parameterize NARR candidate coordinates
2026-06-10 13:50:24 -05:00
aab3c3736c
feat: extend HRRR forecast horizon from 18h to 48h
HRRR publishes f21-f48 at 3-hourly intervals in addition to the
hourly f01-f18 we already fetch. This extends the grid_tasks seed
from 19 rows (1 analysis + 18 forecast) to 29 rows (1 analysis +
18 hourly + 10 3-hourly), and bumps the UI forecast window
constant from 18h to 48h so the map timeline and path calculator
forecast chart automatically show the extended horizon.

No Rust logic changes needed — the pipeline is data-driven and
u8 forecast_hour already handles f48.
2026-06-10 11:36:58 -05:00
d67186176f
fix: resolve all dialyzer type errors across project (46→0 project errors)
Type spec fixes:
- duct_usable_* return boolean→float (delegate to duct_usable_for_band)
- sanitize/1 spec includes :unicode error tuples
- match_delete/1 broadened from :ets.match_spec()
- telemetry_event local type replaces :telemetry.event/0
- wgrib2 parse_lon_val_segment corrected to tuple spec
- preloaded Ecto assoc types in get_mission/get_contact! specs

Unmatched returns:
- _ = prefix on Task.start, Oban.insert, Repo.query!, PubSub.subscribe,
  :ets.new, and if-expression returns across 18 files

Pattern match fixes:
- markdown: restructure acc!=[] guard as direct pattern match
- path_compute/pskr/skewt_location_resolver: remove dead clauses
- calibrate.aprs_144: remove unreachable format_float catch-all
- unused.ex: suppress MapSet.union no_opaque

Also: remove unused unicode_util_compat from mix.lock
2026-06-08 17:51:13 -05:00
aa5c2b4808
chore: update elixir 1.20.0 / erlang 29.0.1 / debian trixie, add libsctp1 2026-06-04 15:41:20 -05:00
ea3033da03
chore: update to elixir 1.20 / otp 29, fix compile warnings
- Update .tool-versions to elixir 1.20.0-otp-29 / erlang 29.0.1
- Update all hex dependencies
- Remove unused aliases in path_compute.ex and rover_planning_live/show.ex
- Fix Oban unique states to include :suspended (use :incomplete group)
2026-06-03 14:34:39 -05:00
ec1529fa6b
fix: propagation overlay broken by iolist→binary crash in encode_binary
Two regressions from the recent performance commits:

1. ScoresController.encode_binary/1 (9cce257d): Enum.reduce produces
   iolists, but <<lats::binary>> requires actual binaries. Caused HTTP
   500 on every /scores/cells request when score files exist on disk,
   making the entire propagation heatmap overlay invisible.

2. ScoreCache.valid_times/1 (15f4175c): Pipe syntax feeds @table as
   the first argument to :ets.foldl/3, creating a 4-argument call
   that crashes at runtime.
2026-06-02 10:44:09 -05:00
e8b143813e
fix: add error handling to unsupervised Task.start calls in application.ex
- Grid cache warm-up and ML model load now wrapped in try/rescue
  so silent failures are logged instead of lost
2026-06-01 15:40:38 -05:00
bca2174f71
chore: ignore .omo/ directory 2026-06-01 15:39:40 -05:00
15f4175cce
perf: fix remaining medium/low severity inefficiencies
- Radio.group_reciprocals: replace O(n²) nested Enum.filter with O(n)
  Enum.group_by using a hash key (sorted station pair + band + hour)
- contact_live/index.ex: cache monthly bars Repo.all(Contact) query
  with 60s TTL
- ScoreCache.valid_times: replace :ets.select with :ets.foldl to
  avoid intermediate list allocation
- Accounts.list_users: add limit(100) to unbounded query
- Accounts.backfill_missing_home_qth: batch individual Oban.insert
  calls into single Oban.insert_all
2026-06-01 15:39:25 -05:00
9cce257db7
perf: fix medium-severity N+1 and inefficiency patterns
- pending_edit_for_user: add preload for [:user, :contact] to avoid
  lazy-loaded N+1 when templates access those associations
- ScoresController.encode_binary: combine 3 Enum.map passes into a
  single Enum.reduce, halving list traversals for score binary encoding
- find_duplicate_contact: push station-pair/grid matching into SQL
  WHERE clause so dedup uses the DB index instead of loading all
  matching rows + Enum.find in Elixir
2026-06-01 15:32:36 -05:00
473c2ab0ec
perf: fix high-severity LiveView blocking queries
- rover_planning_live/show.ex: use pre-loaded @rover_sites assign
  instead of Repo.all(Location) on every keystroke
- rover_locations_live/map.ex: cache locations query with 30s TTL
  (was loading + JSON-encoding all good locations in every mount)
- status_live.ex: consolidate 5 separate stat fetches into single
  cached blob, so PubSub-triggered refreshes hit cache instead of
  running 10+ DB queries each time
2026-06-01 15:31:28 -05:00