Python pskr_mqtt_listen.py:
- Guard against malformed CONNACK/SUBACK/PUBLISH packets
- Fix _s() truthiness: is not None instead of if v (zero SNR was lost)
- EINTR-safe select loop, OOB data handling, VBInt overflow check
- Proper socket cleanup with try/finally + DISCONNECT on shutdown
Elixir:
- Log dropped spot errors in aggregator (was silently discarded)
- Wire retain_scores_window into NotifyListener chain completion
- Recurse sweep_tmp_dir into band/weather_scalars subdirectories
- Rescue recalibrator.run/0 to write failed status row on crash
- Handle nil in fmt_snr/fmt_callsigns (no more nil dB crashes)
- Replace Stream.run with Enum.reduce logging exits in poll_worker
- Handle File.stat race in ms_footprints prune, grid_center nil log
- Fix unused variables, stale comments, missing get_path!/1
Rust:
- Graceful JoinError handling in fetcher/hrdps_fetcher (no more panics)
- round_to_5min returns Option (leap-second safe)
- Acquire/Release ordering on shutdown flags (was Relaxed)
- Parameterized NOTIFY in db.rs, defensive scalar sweep, NaN guard
- OsString::push for tmp naming, clippy fixes
Previously required ≥6-char locators; 4-char fields (~70×100 km)
were dropped. They're less precise for HRRR calibration but still
useful for spot display and coarse path analysis. Lowers the
minimum from 6 to 4 characters.
Co-Authored-By: Claude <noreply@anthropic.com>
Some FT8/Q65 multi-decoder spots omit sender/receiver callsigns
from the JSON payload, but the MQTT topic routing path always
carries them in positions 5 (sc) and 6 (rc):
pskr/filter/v2/<band>/<mode>/<sc>/<rc>/<sl>/<rl>/<sa>/<ra>
Client now extracts callsigns from the topic and threads them
through Aggregator.ingest → Pskr.parse_spot, where they act as
fallback when the JSON sc/rc keys are nil or empty.
JSON payload values still take precedence when present — the topic
is only used as a safety net.
Co-Authored-By: Claude <noreply@anthropic.com>
RepoListener now connects directly to Postgres on port 5432 for
NOTIFY/LISTEN, bypassing pgbouncer (port 6432) which does not
support session-level LISTEN in transaction-pooling mode.
Additionally, connection and LISTEN are moved from init/1 to
handle_continue/2 so the GenServer starts immediately and the
pod boots healthy even when the DB is temporarily unreachable.
Connection failures are retried with exponential backoff
(200ms → 30s max). The notifications process is monitored and
reconnected if it dies.
Falls back to the configured port if port 5432 is unreachable
(e.g. dev/test environments without pgbouncer).
Co-Authored-By: Claude <noreply@anthropic.com>
PSK Reporter tab:
- Fix handle_info(:refresh_spots) to use start_async instead of blocking
- Add require Logger to fix compiler warning
- Make band counts clickable: clicking a band filters table to that band's
last 100 spots; clicking again clears the filter
- Highlight active filter badge, show Clear filter link when filtered
- Update subtitle and empty state dynamically based on filter
- Filter persists across auto-refresh (60s)
- Fix empty state condition to render immediately without async guard
Credo fixes (mix credo --strict now passes):
- weather_map_component.ex: replace @doc false with @impl true (missing spec)
- contact_weather_enqueue_worker.ex: extract hrrr_placeholder_for_contact/3
to reduce nesting depth
- profile_lookup.ex: replace MapSet+then+reject pattern with Enum.filter
to fix both nesting depth and cyclomatic complexity
Test infrastructure fixes:
- Fix Release.migrate/0 to handle repos without migration directories
(AprsRepo in test has no migrations)
- Fix insert_spot helper: add missing 19th param for inserted_at/updated_at
- Fix String.index/2 -> :binary.match for Elixir 1.20 compat
- Fix DateTime.add!/3 -> DateTime.add/3 for Elixir 1.20 compat
- Fix substring matching in limit test (GRID1 matched GRID10)
- Add Process.sleep+render calls for async band_counts in new tests
- Replace impossible 'empty state for filtered band' test with
'switching band filter updates table' test
- All 19 tests pass, mix credo --strict clean
Close 6 cleanup gaps on the shared /data NFS mount:
- HRDPS .hrdps.prop score files: parse_valid_time regex now matches
the .hrdps.prop extension, so these files are found and pruned
- HRDPS scalar dirs (weather_scalars/{iso}.hrdps/): strip .hrdps suffix
before DateTime.from_iso8601 in list_valid_time_dirs
- ScalarFile: prune_older_than was defined but never called from
Propagation.prune_old_scores - now called alongside Scores/Profiles
- Orphan .tmp.* files: sweep all three base dirs (scores/profiles/scalars)
for .tmp.* files left by crashed atomic-write processes
- Building cache (/data/buildings/): add MsFootprints.prune_older_than
with 30-day mtime cutoff
- Canopy cache (/data/canopy/ + staging/): add Canopy.prune_older_than
with 30-day cutoff, cleanup empty staging dir
All cleanup is cutoff-based (older than X time) so if the prune worker
is missed for any period it catches up fully on the next run.
PropagationPruneWorker updated to call all new cleanup functions.
- Replace sequential Enum.each with Task.async_stream for station I/O
- Replace per-row upsert_surface_observation with batch upsert_surface_observations
- Add index on weather_stations(station_type, lat, lon) for nearby_stations
- Add UPPER(station1/2) expression indexes for callsign search
- Batch sync_network ~3K individual Repo.insert calls into one insert_all
- Batch Oban.insert calls in insert_unique into Oban.insert_all
- Log warnings on station elevation Repo.update errors instead of discarding
- Wrap reconcile_mission_paths delete+insert in Repo.transaction
- BeaconLive.Index: targeted stream_insert/delete instead of full re-query+push_patch
- RoverPlanningLive.Show: re-fetch single path instead of re-querying all
- PskrSpotsLive: convert to LiveView streams, add 60s auto-refresh
- ImportConfetti: add missing phx-update=ignore
- accounts.ex: replace if with with-chain in get_user_by_email_and_password
for explicit pattern matching on User struct + valid_password? result.
Collapse nested case into with in create_api_token.
- map_live.ex: extract connected? PubSub subscription block from mount into
named subscribe_if_connected/1 function (cleaner than inline if with _ =).
Deduplicate identical select_time + set_selected_time handlers into shared
handle_set_time/2 helper. Extract toggle_grid + toggle_radar into shared
handle_toggle/3 helper (functions-as-values pattern for assign key).
- CacheTest: fix sweep timing race by using negative TTL (-1)
instead of positive TTL (1) for already-expired entries
- ScoreCache: replace ETS match-spec DateTime comparisons with
:ets.foldl + DateTime.compare — DateTime structs are maps in
Elixir >= 1.15 and ETS can't compare maps with :< / :> guards
- Accounts: drop unsupported returning: true on delete_all,
return [] for expired tokens list
- Backtest: catch ArgumentError from String.to_existing_atom
for unknown feature names, preserving the helpful Mix.Error
- ContactLive IndexTest: invalidate monthly_bars cache before
assertion so test data is visible
- RoverLocationsLive MapTest: invalidate cached points before
assertion
- StatusLiveTest: add DB cleanup in setup to reduce test
interference; parameterize NARR candidate coordinates
HRRR publishes f21-f48 at 3-hourly intervals in addition to the
hourly f01-f18 we already fetch. This extends the grid_tasks seed
from 19 rows (1 analysis + 18 forecast) to 29 rows (1 analysis +
18 hourly + 10 3-hourly), and bumps the UI forecast window
constant from 18h to 48h so the map timeline and path calculator
forecast chart automatically show the extended horizon.
No Rust logic changes needed — the pipeline is data-driven and
u8 forecast_hour already handles f48.
- Update .tool-versions to elixir 1.20.0-otp-29 / erlang 29.0.1
- Update all hex dependencies
- Remove unused aliases in path_compute.ex and rover_planning_live/show.ex
- Fix Oban unique states to include :suspended (use :incomplete group)
Two regressions from the recent performance commits:
1. ScoresController.encode_binary/1 (9cce257d): Enum.reduce produces
iolists, but <<lats::binary>> requires actual binaries. Caused HTTP
500 on every /scores/cells request when score files exist on disk,
making the entire propagation heatmap overlay invisible.
2. ScoreCache.valid_times/1 (15f4175c): Pipe syntax feeds @table as
the first argument to :ets.foldl/3, creating a 4-argument call
that crashes at runtime.
- Radio.group_reciprocals: replace O(n²) nested Enum.filter with O(n)
Enum.group_by using a hash key (sorted station pair + band + hour)
- contact_live/index.ex: cache monthly bars Repo.all(Contact) query
with 60s TTL
- ScoreCache.valid_times: replace :ets.select with :ets.foldl to
avoid intermediate list allocation
- Accounts.list_users: add limit(100) to unbounded query
- Accounts.backfill_missing_home_qth: batch individual Oban.insert
calls into single Oban.insert_all
- pending_edit_for_user: add preload for [:user, :contact] to avoid
lazy-loaded N+1 when templates access those associations
- ScoresController.encode_binary: combine 3 Enum.map passes into a
single Enum.reduce, halving list traversals for score binary encoding
- find_duplicate_contact: push station-pair/grid matching into SQL
WHERE clause so dedup uses the DB index instead of loading all
matching rows + Enum.find in Elixir
- NexradCache: replace :ets.tab2list() with foldl to avoid ~1.3 GB heap
allocation on full cache (OOM risk)
- GridCache: replace :ets.select full-table copies with foldl in
ets_latest_valid_time and ets_prune_keep_latest
- Weather.backfill_hrrr_batch: batch N individual UPDATEs into single
VALUES-based SQL statement eliminating ~500 round-trips per batch
- Add 9 missing database indexes across 6 tables for hot-path queries
(contacts, contact_edits, hrrr_profiles, hrrr_native_profiles,
grid_tasks, rover_mission_paths, fixed_stations, beacons)
- Mailer: apply_defaults/1 sets From and Reply-To headers
- AboutLive: content rendering, empty stats, and contact count display
- Fix has_many :beacons association (wrong module path)
- Fix router pipeline order: serve_markdown before accepts,
after secure headers so markdown responses get security headers
- Fix notify_listener_test Process.sleep regression
- Update findings.md test coverage gap status
- grid.ex: memoize conus_points/0 with module attribute (was ~94k recompute per call)
- path_compute.ex: reduce 6 list traversals to single Enum.reduce pass
- radio.ex + application.ex: fix unused variable e and redundant case warnings
- priv: add 5 partial indexes on (qso_timestamp) for enrichment queries (weather, hrrr,
terrain, iemre, radar) matching the WHERE status IN ('pending','failed') AND pos1 IS NOT NULL
pattern used by contacts_needing_enrichment
- findings.md: mark all performance hotspots and corresponding items as fixed
- Replace deprecated xref:exclude with elixirc_options in vendor/oban_pro
- Remove unused require Logger from 8 files
- Pin bitstring size variables with ^ in simple_packing, wgrib2,
complex_packing, section, nexrad_client, mqtt, and radar worker
- Remove dead defp clauses (format/1 nil, depression/2 nil)
- Rename Buildings.Parser type record/0 -> building_record/0 to avoid
overriding built-in type
- Remove redundant catch-all in candidate_detail.ex
- Simplify contact_live/show.ex conditional based on type narrowing
- Fix DateTime.from_iso8601 return pattern in vendor/oban_web
- Upgrade phoenix_live_view to 1.1.31
- Drop --warnings-as-errors from precommit alias; known false positives
from @after_verify-generated code in Elixir 1.20.0-rc.6 + PLV 1.1.x
- Add credo --strict to precommit as replacement static-analysis gate
Replace the two-script Python pipeline (analysis report + Elixir-source
emitter) with a single `scripts/recalibrate.py` that fits per-band
weights from PSKR spot density (VHF/UHF) and contacts↔HRRR correlations
(microwave), writing `priv/algo/band_weights.json` as a machine-readable
artifact. A new Elixir BandWeights module loads this JSON once via
`:persistent_term` cache; BandConfig.weights/1 consults it before falling
back to in-source overrides or global defaults. The script never touches
Elixir source — recalibration is now `python3 scripts/recalibrate.py`
followed by an app restart.
Show on /weather where the detected duct geometry traps each microwave
band. Overview mode bins cells by their lowest trapped frequency
(Bean & Dutton cutoff); band-picker mode masks cells whose duct doesn't
reach the selected band. Cutoff is pre-computed per cell in both
writers (Elixir derive + Rust derive_row reads best_duct_freq_ghz from
CellValues), persisted to ScalarFile, and shipped to the JS hook via
the existing binary cell pack. Also cleans up six pre-existing
length/1 credo warnings in unrelated test files.
POST /api/v1/beacon-monitor/measurements accepts one measurement per
integration window from a propmonitor client, authenticated by the
BeaconMonitor token. Records noise floor, signal peak/avg dBFS, SNR,
signal-active-fraction, gain, and frequency for later correlation
against weather and propagation scores.
Beacon UUID must resolve to an approved + on-the-air beacon (404
otherwise — the client drops 404s without retry). Retries with the
same (monitor, beacon, measured_at) are idempotent: a unique index
makes the second insert a 409. Every accepted upload stamps the
monitor's last_seen_at — measurements *are* the heartbeat.
MonitorAuth is a separate plug from the user API-token Auth plug;
monitors are not users. The rate limiter buckets each monitor by id so
retry storms after a 5xx don't burn the shared anon-IP bucket.
PathCompute.compute/5 now accepts an :on_progress callback and emits
9 named stages (resolve → terrain → HRRR grid → atmospheric → sounding
→ scoring → budget → forecast → ionosphere). PathLive runs the compute
in start_async, streams {:compute_progress, step, total, label}
messages from the callback, and renders the current stage + step/total
in the disabled button alongside a daisyUI progress bar — replacing
the static "Computing..." while the multi-second pipeline runs.
handle_async covers :ok, :error, and {:exit, _} (last logs per the
CLAUDE.md async-error rule). Existing PathLive tests that asserted on
result content after live() were switched to render_async(lv) so they
wait for the task; new tests cover the progress callback ordering and
the in-flight label rendering.
- /u/:callsign profile no longer leaks private contacts or pending beacons;
Radio.list_contacts_for_user/2 and Beacons.list_beacons_for_user/2 now
filter by viewer (owner/admin see everything, others see only public).
- Radio.create_contact/2 places user_id on the struct before
submission_changeset so validate_user_or_email/1 accepts authenticated
submissions that omit submitter_email.
- valkey_test.exs runs async: false; its setup mutates global Application
env and Process registry, which otherwise crashed concurrent ConnCase
GridCache.clear/0 calls (Mox.UnexpectedCallError on SCAN).
- Radio.normalize_proposed/1 now Map.take/2 the allowed-edit keys
before normalization. Owner / admin / pending-edit submit paths all
funnel through this boundary, so a crafted form can no longer
mass-assign user_id, flagged_invalid, inserted_at, etc. on a
contact via String.to_existing_atom -> Ecto.Changeset.change.
- Accounts.revoke_api_token/2 rescues Ecto.Query.CastError and
returns {:error, :not_found} so a malformed UUID in
DELETE /api/v1/me/api-tokens/:id renders the API's clean 404
problem+json instead of a 500.
- ProfilesFile.read_etf decodes with :erlang.binary_to_term(bin, [:safe]).
Defense-in-depth against tampered on-disk profiles (atom-table
exhaustion via untrusted ETF).
- Regression tests: contact_edit_test.exs covers the rejected
mass-assignment fields; accounts_api_token_test.exs covers the
malformed-UUID path.
- Beacon detail endpoints (LiveView + REST API) now hide unapproved
beacons from anonymous and unauthorized viewers; only the submitter
and admins can see pending records before approval. Adds
Beacons.get_visible_beacon/2 with scope-aware checks.
- API contact pagination now honors per_page end-to-end.
Radio.list_contacts/1 accepts :per_page and clamps to 200.
- API rate limiter: ETS table is now owned by a long-lived Sweeper
GenServer (won't die with a request task); Sweeper periodically
prunes expired-window rows to bound memory; init_table/0 race is
rescued.
- /scores/cells and /weather/cells: add per-IP rate limiting and a
shared GridBounds clamp/413 guard so global / oversized viewports
no longer drive unbounded binary responses.
- NEXRAD PNG unfilter (sub/up/average/paeth): replace acc++[byte]
+ Enum.at(acc, idx-bpp) with O(n) binary recursion. Decode time
for the 12200x5400 n0q frame goes from quadratic to linear.
- LiveTableFooter.parse_page and ScoresFile.fetch_bound: switch
String.to_integer/String.to_float to Integer.parse/Float.parse,
fall back to defaults instead of raising.
- PathLive and MapLive band-event handlers: replace
String.to_integer(params["band"]) with parse_int / parse_band_param
so a non-numeric band parameter no longer crashes the LiveView.
Adds a separate API-token section under /users/settings (distinct from
the beacon-monitor token list, since API tokens grant full account
access). The plaintext is surfaced exactly once via flash on creation;
only the SHA-256 hash is persisted, so revocation is the only path back
if the user loses it.
Also fixes the openapi.yaml link on /docs/api: the relative path
resolved to /docs/openapi.yaml from a no-trailing-slash URL and 404'd.
Two issues made docs/api hard to read:
1. The site's custom Markdown parser only recognized `- ` bullets,
so the README's `* ` bullets rendered as a single run-on paragraph
with literal asterisks. Extended the parser (and tests) to accept
the full CommonMark bullet set: `-`, `*`, `+`.
2. The shared .markdown-content container is 88rem wide. Comfortable
for /algo's wide tables but uncomfortable for monospace prose,
which prefers ~80ch. Added a .api-docs modifier class on the
/docs/api page that drops max-width to 60rem, allows table
cells to wrap, and slightly downsizes headings.
Adds bearer-token authenticated REST API at /api/v1 covering every
action a non-admin user can perform on the website: contact + beacon
submission, beacon-monitor management, propagation queries, profile
read/update, and self-service API token issuance/revocation.
Security: SHA-256-hashed bearer tokens (mwp_ prefix, plaintext shown
once at creation), RFC 9457 problem+json error responses, RFC 9651
RateLimit-* headers backed by an ETS bucket (600/min per token,
60/min per anonymous IP, 30/min on /auth/tokens), private-contact
filtering by viewer.
Docs at docs/api/README.md (prose reference) and docs/api/openapi.yaml
(OpenAPI 3.1 spec covering every endpoint, response, and schema).
Tests: 124 new tests across schema, plug, error renderer, rate
limiter, fallback, and every controller. 16/17 API modules at 100%
line coverage; FallbackController at 87.5% (one defmodule line, an
Erlang-cover artifact for action_fallback-only modules).
