- Update .tool-versions to elixir 1.20.0-otp-29 / erlang 29.0.1
- Update all hex dependencies
- Remove unused aliases in path_compute.ex and rover_planning_live/show.ex
- Fix Oban unique states to include :suspended (use :incomplete group)
Two regressions from the recent performance commits:
1. ScoresController.encode_binary/1 (9cce257d): Enum.reduce produces
iolists, but <<lats::binary>> requires actual binaries. Caused HTTP
500 on every /scores/cells request when score files exist on disk,
making the entire propagation heatmap overlay invisible.
2. ScoreCache.valid_times/1 (15f4175c): Pipe syntax feeds @table as
the first argument to :ets.foldl/3, creating a 4-argument call
that crashes at runtime.
- Radio.group_reciprocals: replace O(n²) nested Enum.filter with O(n)
Enum.group_by using a hash key (sorted station pair + band + hour)
- contact_live/index.ex: cache monthly bars Repo.all(Contact) query
with 60s TTL
- ScoreCache.valid_times: replace :ets.select with :ets.foldl to
avoid intermediate list allocation
- Accounts.list_users: add limit(100) to unbounded query
- Accounts.backfill_missing_home_qth: batch individual Oban.insert
calls into single Oban.insert_all
- pending_edit_for_user: add preload for [:user, :contact] to avoid
lazy-loaded N+1 when templates access those associations
- ScoresController.encode_binary: combine 3 Enum.map passes into a
single Enum.reduce, halving list traversals for score binary encoding
- find_duplicate_contact: push station-pair/grid matching into SQL
WHERE clause so dedup uses the DB index instead of loading all
matching rows + Enum.find in Elixir
- rover_planning_live/show.ex: use pre-loaded @rover_sites assign
instead of Repo.all(Location) on every keystroke
- rover_locations_live/map.ex: cache locations query with 30s TTL
(was loading + JSON-encoding all good locations in every mount)
- status_live.ex: consolidate 5 separate stat fetches into single
cached blob, so PubSub-triggered refreshes hit cache instead of
running 10+ DB queries each time
- NexradCache: replace :ets.tab2list() with foldl to avoid ~1.3 GB heap
allocation on full cache (OOM risk)
- GridCache: replace :ets.select full-table copies with foldl in
ets_latest_valid_time and ets_prune_keep_latest
- Weather.backfill_hrrr_batch: batch N individual UPDATEs into single
VALUES-based SQL statement eliminating ~500 round-trips per batch
- Add 9 missing database indexes across 6 tables for hot-path queries
(contacts, contact_edits, hrrr_profiles, hrrr_native_profiles,
grid_tasks, rover_mission_paths, fixed_stations, beacons)
- Mailer: apply_defaults/1 sets From and Reply-To headers
- AboutLive: content rendering, empty stats, and contact count display
- Fix has_many :beacons association (wrong module path)
- Fix router pipeline order: serve_markdown before accepts,
after secure headers so markdown responses get security headers
- Fix notify_listener_test Process.sleep regression
- Update findings.md test coverage gap status
- grid.ex: memoize conus_points/0 with module attribute (was ~94k recompute per call)
- path_compute.ex: reduce 6 list traversals to single Enum.reduce pass
- radio.ex + application.ex: fix unused variable e and redundant case warnings
- priv: add 5 partial indexes on (qso_timestamp) for enrichment queries (weather, hrrr,
terrain, iemre, radar) matching the WHERE status IN ('pending','failed') AND pos1 IS NOT NULL
pattern used by contacts_needing_enrichment
- findings.md: mark all performance hotspots and corresponding items as fixed
- Replace deprecated xref:exclude with elixirc_options in vendor/oban_pro
- Remove unused require Logger from 8 files
- Pin bitstring size variables with ^ in simple_packing, wgrib2,
complex_packing, section, nexrad_client, mqtt, and radar worker
- Remove dead defp clauses (format/1 nil, depression/2 nil)
- Rename Buildings.Parser type record/0 -> building_record/0 to avoid
overriding built-in type
- Remove redundant catch-all in candidate_detail.ex
- Simplify contact_live/show.ex conditional based on type narrowing
- Fix DateTime.from_iso8601 return pattern in vendor/oban_web
- Upgrade phoenix_live_view to 1.1.31
- Drop --warnings-as-errors from precommit alias; known false positives
from @after_verify-generated code in Elixir 1.20.0-rc.6 + PLV 1.1.x
- Add credo --strict to precommit as replacement static-analysis gate
Replace the two-script Python pipeline (analysis report + Elixir-source
emitter) with a single `scripts/recalibrate.py` that fits per-band
weights from PSKR spot density (VHF/UHF) and contacts↔HRRR correlations
(microwave), writing `priv/algo/band_weights.json` as a machine-readable
artifact. A new Elixir BandWeights module loads this JSON once via
`:persistent_term` cache; BandConfig.weights/1 consults it before falling
back to in-source overrides or global defaults. The script never touches
Elixir source — recalibration is now `python3 scripts/recalibrate.py`
followed by an app restart.
Show on /weather where the detected duct geometry traps each microwave
band. Overview mode bins cells by their lowest trapped frequency
(Bean & Dutton cutoff); band-picker mode masks cells whose duct doesn't
reach the selected band. Cutoff is pre-computed per cell in both
writers (Elixir derive + Rust derive_row reads best_duct_freq_ghz from
CellValues), persisted to ScalarFile, and shipped to the JS hook via
the existing binary cell pack. Also cleans up six pre-existing
length/1 credo warnings in unrelated test files.
format_number rounds to 1 decimal, so the tiny per-km coefficients
(o2_db_km: 0.007 at 10 GHz, gas rate ~0.007 dB/km) all displayed as
'0.0'. Added format_small with 4-decimal precision for those fields.
POST /api/v1/beacon-monitor/measurements accepts one measurement per
integration window from a propmonitor client, authenticated by the
BeaconMonitor token. Records noise floor, signal peak/avg dBFS, SNR,
signal-active-fraction, gain, and frequency for later correlation
against weather and propagation scores.
Beacon UUID must resolve to an approved + on-the-air beacon (404
otherwise — the client drops 404s without retry). Retries with the
same (monitor, beacon, measured_at) are idempotent: a unique index
makes the second insert a 409. Every accepted upload stamps the
monitor's last_seen_at — measurements *are* the heartbeat.
MonitorAuth is a separate plug from the user API-token Auth plug;
monitors are not users. The rate limiter buckets each monitor by id so
retry storms after a 5xx don't burn the shared anon-IP bucket.
PathCompute.compute/5 now accepts an :on_progress callback and emits
9 named stages (resolve → terrain → HRRR grid → atmospheric → sounding
→ scoring → budget → forecast → ionosphere). PathLive runs the compute
in start_async, streams {:compute_progress, step, total, label}
messages from the callback, and renders the current stage + step/total
in the disabled button alongside a daisyUI progress bar — replacing
the static "Computing..." while the multi-second pipeline runs.
handle_async covers :ok, :error, and {:exit, _} (last logs per the
CLAUDE.md async-error rule). Existing PathLive tests that asserted on
result content after live() were switched to render_async(lv) so they
wait for the task; new tests cover the progress callback ordering and
the in-flight label rendering.
- /u/:callsign profile no longer leaks private contacts or pending beacons;
Radio.list_contacts_for_user/2 and Beacons.list_beacons_for_user/2 now
filter by viewer (owner/admin see everything, others see only public).
- Radio.create_contact/2 places user_id on the struct before
submission_changeset so validate_user_or_email/1 accepts authenticated
submissions that omit submitter_email.
- valkey_test.exs runs async: false; its setup mutates global Application
env and Process registry, which otherwise crashed concurrent ConnCase
GridCache.clear/0 calls (Mox.UnexpectedCallError on SCAN).
- Radio.normalize_proposed/1 now Map.take/2 the allowed-edit keys
before normalization. Owner / admin / pending-edit submit paths all
funnel through this boundary, so a crafted form can no longer
mass-assign user_id, flagged_invalid, inserted_at, etc. on a
contact via String.to_existing_atom -> Ecto.Changeset.change.
- Accounts.revoke_api_token/2 rescues Ecto.Query.CastError and
returns {:error, :not_found} so a malformed UUID in
DELETE /api/v1/me/api-tokens/:id renders the API's clean 404
problem+json instead of a 500.
- ProfilesFile.read_etf decodes with :erlang.binary_to_term(bin, [:safe]).
Defense-in-depth against tampered on-disk profiles (atom-table
exhaustion via untrusted ETF).
- Regression tests: contact_edit_test.exs covers the rejected
mass-assignment fields; accounts_api_token_test.exs covers the
malformed-UUID path.
- Beacon detail endpoints (LiveView + REST API) now hide unapproved
beacons from anonymous and unauthorized viewers; only the submitter
and admins can see pending records before approval. Adds
Beacons.get_visible_beacon/2 with scope-aware checks.
- API contact pagination now honors per_page end-to-end.
Radio.list_contacts/1 accepts :per_page and clamps to 200.
- API rate limiter: ETS table is now owned by a long-lived Sweeper
GenServer (won't die with a request task); Sweeper periodically
prunes expired-window rows to bound memory; init_table/0 race is
rescued.
- /scores/cells and /weather/cells: add per-IP rate limiting and a
shared GridBounds clamp/413 guard so global / oversized viewports
no longer drive unbounded binary responses.
- NEXRAD PNG unfilter (sub/up/average/paeth): replace acc++[byte]
+ Enum.at(acc, idx-bpp) with O(n) binary recursion. Decode time
for the 12200x5400 n0q frame goes from quadratic to linear.
- LiveTableFooter.parse_page and ScoresFile.fetch_bound: switch
String.to_integer/String.to_float to Integer.parse/Float.parse,
fall back to defaults instead of raising.
- PathLive and MapLive band-event handlers: replace
String.to_integer(params["band"]) with parse_int / parse_band_param
so a non-numeric band parameter no longer crashes the LiveView.
Replace the markdown-rendered /docs/api page with a structured
LiveView modelled on the Tailwind UI Protocol template — fixed
left sidebar with grouped section nav, hero with metadata dl,
two-column rows where prose sits next to a sticky code sample,
and endpoint cards with color-coded HTTP method tags.
Adds a separate API-token section under /users/settings (distinct from
the beacon-monitor token list, since API tokens grant full account
access). The plaintext is surfaced exactly once via flash on creation;
only the SHA-256 hash is persisted, so revocation is the only path back
if the user loses it.
Also fixes the openapi.yaml link on /docs/api: the relative path
resolved to /docs/openapi.yaml from a no-trailing-slash URL and 404'd.
Two issues made docs/api hard to read:
1. The site's custom Markdown parser only recognized `- ` bullets,
so the README's `* ` bullets rendered as a single run-on paragraph
with literal asterisks. Extended the parser (and tests) to accept
the full CommonMark bullet set: `-`, `*`, `+`.
2. The shared .markdown-content container is 88rem wide. Comfortable
for /algo's wide tables but uncomfortable for monospace prose,
which prefers ~80ch. Added a .api-docs modifier class on the
/docs/api page that drops max-width to 60rem, allows table
cells to wrap, and slightly downsizes headings.
Adds bearer-token authenticated REST API at /api/v1 covering every
action a non-admin user can perform on the website: contact + beacon
submission, beacon-monitor management, propagation queries, profile
read/update, and self-service API token issuance/revocation.
Security: SHA-256-hashed bearer tokens (mwp_ prefix, plaintext shown
once at creation), RFC 9457 problem+json error responses, RFC 9651
RateLimit-* headers backed by an ETS bucket (600/min per token,
60/min per anonymous IP, 30/min on /auth/tokens), private-contact
filtering by viewer.
Docs at docs/api/README.md (prose reference) and docs/api/openapi.yaml
(OpenAPI 3.1 spec covering every endpoint, response, and schema).
Tests: 124 new tests across schema, plug, error renderer, rate
limiter, fallback, and every controller. 16/17 API modules at 100%
line coverage; FallbackController at 87.5% (one defmodule line, an
Erlang-cover artifact for action_fallback-only modules).
Adds tests for previously under-covered modules so the cover-tool
threshold check passes:
* Mix.Tasks.Prop.Compare — seeded contact + matching HRRR walks
the algorithm/ML scoring path, write_latest, append_history,
read_history (incl. the unparseable-line arm), and the per-band
summary loop.
* Mix.Tasks.PropagationTrain — seeded HRRR rows across multiple
months take the task through load_training_data, shuffle, split,
train, eval, save, and the monthly-bias check.
* Mix.Tasks.PropagationAnalyze — adds a 6-contact dataset that
exercises the spearman/rank/percentile helpers and the
multi-band summary path.
* Mix.Tasks.Unused — smoke tests run/1 with no flags,
--skip-external, and --verbose.
* Mix.Tasks.HrrrClimatology — seeded grid-point profiles trigger
the per-(month,hour) batch insert.
* Microwaveprop.Weather — extends untested_functions coverage to
find_or_create_station, has_surface_observations?,
station_day_covered?, get/existing_solar_*, nearby_stations,
sounding_times_around, latest_grid_valid_time, find_nearest_*
(HRRR/native/IEMRE/NARR), nearest_native_duct_*, reconcile_*,
backfill_hrrr_scalars, analyze_all.
* Microwaveprop.Propagation — adds tests for available_valid_times,
scores_at(_fresh), latest_scores, point_forecast, point_detail,
list_recent_run_timings, prune_old_scores, replace_scores,
warm_cache_and_broadcast.
* Microwaveprop.Backtest.Features — adds duct_usable_*ghz alias
delegations.
* Microwaveprop.Weather.IemRateLimiter — covers the is_pid clause
of registered?/1 with a live PID.
* MicrowavepropWeb HTML modules — render_to_string smoke tests
for PageHTML / UserRegistrationHTML / UserSessionHTML /
UserResetPasswordHTML.
Also pins `test_coverage: [summary: [threshold: 85]]` in mix.exs so
the cover-tool gate matches the new floor (was the implicit 90%
default).
Total goes from 82.77% → 85.06%.
Rust cell_to_profile_entry now writes surface_refractivity and
min_refractivity_gradient at the cell level so Elixir consumers have a
guaranteed fallback when SoundingParams.derive hits levels without
dewpoint. The SkewT build_profile_assigns falls back to these pre-computed
cell keys when the derived map has nil for those fields.
MicrowavepropWeb.Telemetry's :telemetry_poller fires
dispatch_oban_queue_depth/0 every 30 s. In test env it runs outside
any Ecto sandbox owner, so its `Repo.all` against oban_jobs holds a
pool connection long enough to starve LiveView `render_async` tests
with 100 ms async windows — repro'd as flaky SkewtLive failures.
Same shape as the PromEx Oban plugin we already disable in test via
:prom_ex_include_oban_plugin. Add :start_telemetry_poller (defaults
to true) and toggle it off in config/test.exs. Production keeps the
30-second per-(queue,state) gauge.
- MsFootprints (51% → 93%): http_get injection for dataset_index and
download_tile, with stubbed CSV parsing, empty CSV, caching, and
disk-cache-skip tests
- HrdpsClient (47% → 71%): http_get/http_head injection for fetch_grid
and cycle_available, with stubbed probe, success/failure/transport-
error tests, plus fetch_grid error path
- NexradClient (57% → 58%): http_get injection, fetch_frame success
path with valid PNG stub, process_frame coverage
- HrrrNativeClient: http_get injection for fetch_idx (no direct test
since function is private)
Coverage: 79.43% → 79.68% (need 0.32% more)
- Fix production bug in valkey_fetch_point: Map.get(chunk_map, {lat, lon})
was looking up lat/lon in the outer map keyed by chunk tuple, not the
inner cells map. Added chunk_key unwrap before lat/lon lookup.
- DataStubAdapter returns real chunk data for fetch/fetch_bounds/fetch_point
- Tests cover: fetch with decoded rows, fetch_bounds filtering, fetch_point
with chunk unwrap, put with large row sets, claim_fill/release_fill
Coverage: 79.30% → 79.43% (GridCache 74% → 84%)
- PathCompute: compute_loss_budget/5 and compute_power_budget/2 made
public with @doc false; property tests covering fspl, o2, h2o, rain,
diffraction across all 13 known bands + varying distances
- Viewshed: property tests for destination_point back-bearing round-trip
and east-west bearing at equator (2 new properties)
Coverage: 79.07% → 79.06% (PathCompute 60→63%)
- Create Microwaveprop.Valkey.Adapter behaviour with command/3 + pipeline/3
- Create Microwaveprop.Valkey.RedixAdapter as the production impl
- Inject adapter via Application config; mock with Mox in tests
- Cover all Valkey operations: get, mget, set, mset_with_ttl, zadd,
zrevrange, zrangebyscore, zrem, del, scan_match (multi-cursor)
- Full round-trip tests for encode/decode, error paths, edge cases
Coverage: 78.93% → 79.07%
The producer-only path left older hours stuck at 0% HRRR coverage
forever once their fetch tasks drained: the rolling-window cron only
re-passes the prior hour, so manual lookback runs (or any
out-of-window hour) wrote NULL samples + enqueued tasks, then never
re-ran to UPSERT the now-landed HRRR data.
Fix: split the responsibility cleanly.
* CalibrationSampler.build_for_hour/1 now returns
%{upserted: int, missing_hrrr_cells: int} so callers know whether
the producer enqueued anything (i.e. whether a follow-up makes
sense). Spec-tightened with a new @type result.
* PskrCalibrationWorker, after each hour's sampler pass, schedules a
follow-up of itself for that exact hour 10 min later when missing
> 0. The follow-up carries args %{"hour_utc" => ..., "_follow_up"
=> true}; the sentinel prevents follow-ups from chaining further
follow-ups (rolling-window cron is the safety net).
Tests:
* Updated 4 existing sampler tests to the new map return shape.
* Moved the follow-up assertions from the sampler suite into the
worker suite where they belong (sampler is now scheduling-free).
* Added 3 worker tests: schedules-when-missing, no-schedule-when-
covered, follow-up-doesn't-chain.
3313 tests + 228 properties, 0 failures.
Two performance fixes the property tests pinned down:
* CalibrationSampler.build_for_hour/1 walked every cell through
nearest_hrrr/3 twice — once in the producer and again in
build_sample. Now computed once into a (band, lat, lon) → hrrr_or_nil
map, halving the O(cells × profiles) scan (~10M comparisons saved
per fire at typical sizes).
* PartitionManager.ensure_quarterly_partitions/2 issued one
pg_inherits scan per (parent × lookahead). Refactored to one scan
per parent with in-memory coverage check across all quarters.
Property tests for both modules:
* PartitionManager: tile-coverage invariant (no gaps, no overlaps),
exact 3-month bounds, name format, multi-call idempotence,
multi-parent independence — caught a real NaiveDateTime sort bug
in the original test helpers. Default Enum.sort_by/2 falls back to
term comparison on NaiveDateTime; now uses NaiveDateTime as the
comparator module.
* CalibrationSampler: enqueued points = unique missing midpoints,
empty enqueue when fully covered, sample row count invariant under
HRRR availability.
Saved operational gotchas to CLAUDE.md (Oban leader election spans
all app=prop pods, kubectl exec deploy/prop ambiguity, half-year
partition coexistence, HRRR f000 publish lag).
3310 tests + 228 properties, 0 failures.
Adds the test that verifies the full producer→drain→re-pass loop:
sample written with NULL HRRR fields + fetch task enqueued, then once
an HRRR profile lands, the next sampler pass UPSERTs the same
calibration_samples row (id preserved) with non-NULL weather data.
Also tightens dialyzer surface:
* PartitionManager defines a `result()` typedoc and uses it on both
public functions instead of inlined tuple types.
* PartitionMaintenanceWorker.perform/1 + the lookahead/1 helper get
explicit specs.
* CalibrationSampler.enqueue_missing_hrrr/3 gets a spec covering its
group_by-shape input map and HRRR profile list.
`mix precommit` clean (3310 tests, 0 failures). Local `mix dialyzer`
hits an OTP 28.4 vs 28.5 toolchain mismatch unrelated to this change.
Partition list for hrrr_profiles + hrdps_profiles was hand-edited in
priv/repo/structure.sql; if no human extended it, writes for the next
quarter would silently fail at the worker level (Rust hrrr-point-worker
marks tasks failed; the calibration pipeline silently joins NULLs).
Adds:
* Microwaveprop.PartitionManager — runtime DDL helper. Quarter math via
a single integer index (year*4 + quarter) for rollover safety.
Coverage check via pg_inherits before CREATE so a quarterly target
inside an existing wider partition (some 2027 ones are half-year)
is treated as :exists rather than colliding.
* Microwaveprop.Workers.PartitionMaintenanceWorker — daily cron at
03:15 UTC, 4-quarter lookahead. Idempotent; cheap when nothing's
missing. lookback_quarters arg lets ops widen for catch-up.
Tests cover the create + idempotent re-run + name format + year
rollover paths via a synthetic parent created inside the sandbox
transaction (no DDL leaks into the shared test DB).
The CalibrationSampler joined PSKR midpoints against hrrr_profiles via
nearest-within-window, but post-Phase-3 Stream C nothing bulk-populates
the grid — hrrr_profiles only sees per-QSO point fetches from the Rust
hrrr-point-worker. Result: 0 / 16,042 prod samples had HRRR data.
Producer pattern: when a cell has no profile in the lookahead window,
enqueue a row into hrrr_fetch_tasks at the cell's HRRR-grid-rounded
midpoint. The Rust worker drains it; the next 5-min rolling-window
pass UPSERTs the same sample with non-NULL weather fields. Idempotent
with the existing on-conflict design — no extra worker, no churn on
cells already covered, multiple bands sharing a midpoint dedupe to a
single fetch via Enum.uniq + jsonb point-union in HrrrPointEnqueuer.
The polylines were sourcing from --band-* CSS vars (a separate, theme-aware
design palette) while the legend swatches rendered from the Elixir @band_colors
map, so the two diverged for every band — e.g. 10 GHz showed lime on the map
and emerald in the legend. Emit `data-band-color` on each band checkbox and
have the JS hook read from there, making the legend the single source of truth.
Drops the now-redundant theme observer / restyleLines plumbing.
