Reworks the post-fetch half of the propagation pipeline. Fetch and GRIB2
decode were already cheap — measured against a live HRRR cycle, all 39
pressure messages decode via `wgrib2 -lola` in 0.29 s and the 31 MB
byte-range fetch takes ~3 s — so nothing here touches the decoder. All the
cost was downstream.
Also fixes a broken NOTIFY that made every chain step run up to 5 times.
pg_notify
`NOTIFY propagation_ready, $1` is a Postgres syntax error: NOTIFY is a
utility statement whose payload must be a literal, so a bind raises
42601. It shared a transaction with the `status='done'` UPDATE, so every
successful step rolled back, stayed 'running', and was requeued by
reclaim_stale_running up to @max_reclaim_attempts times. Elixir's
NotifyListener never fired either, so ScoreCache warm and the
"propagation:updated" fan-out were dead.
FieldGrid
A decoded grid was HashMap<(i32,i32), HashMap<Arc<str>, f32>> — a dense
rectangular grid stored as ~95k nested hash maps, costing ~4.6M inserts
on decode, ~3.7M on merge and ~14M lookups across three derivation
passes. wgrib2 -lola already emits one dense row-major f32 block per
message, so keep it: dense per-message planes, names hashed once per
grid into plane ids, NaN as the missing sentinel. This is what forced
PROP_GRID_RS_PARALLELISM=1 under a 3Gi limit.
Fused pass
Three 95k-cell derivation passes plus 23 band-major scoring passes over
a staged Vec<(f64,f64,Conditions,BandInvariants)> (~19MB re-streamed 23
times) collapse into one pass: levels extracted once per cell, all 23
bands scored while the cell is hot, scores accumulated cell-major so
rayon chunks own disjoint slices. Scores land straight in the dense
score-file body — no ScorePoint scatter.
.pgrid
The profile artifact was an rmpv tree plus gzip -9, written 30x an hour,
and ProfilesFile.read_point/3 gunzipped and unpacked the entire 95k-cell
file to return one cell on every map click and Skew-T load. Replaced
with a dense cell-major f32 record array carrying a self-describing
field table. Elixir reads it via :file.pread; .mp.gz and .etf.gz remain
readable so files written before this drain out of the 48h window.
Measured on a full CONUS grid (95,073 cells x 48 planes x 23 bands):
derive + score + build artifacts 0.022 s
profile write 3.957 s -> 0.006 s (22.0 MB -> 22.4 MB on disk)
single-cell read whole-file decode -> 0.5 us
23 score files 0.003 s
Also
- hrrr_points: batched UNNEST upsert replacing one awaited INSERT per
point. Keeps ON CONFLICT DO UPDATE — the PSKR sampler's two-pass loop
depends on it.
- fetcher: real semaphore capping in-flight ranges at
MAX_PARALLEL_RANGES, which the comment claimed but the code did not do
(it spawned all 27 while the connection pool was sized for 8).
- metrics: per-stage histogram. Only chain-step and decode durations
were instrumented, which is why the write cost stayed invisible.
- profiles_file: parse_valid_time anchors on the known extension set, so
sibling-suffixed names like <iso>.hrdps.prop no longer parse as
<iso>.hrdps and vanish from prune and list operations.
- PROP_GRID_RS_PARALLELISM 1 -> 3. Memory limit held at 3Gi until RSS is
observed at the new parallelism.
- cargo fmt over the crate; worker.rs, hrdps_fetcher.rs and nexrad.rs
were already unformatted at HEAD and the pre-commit hook gates on it.
HRDPS still runs at 0.5 degrees. wgrib2 -lola scales linearly in output
points on rotated lat/lon (12.5 s wall, 202 s CPU for one message at
0.125 degrees) because it has no inverse projection for those grids; a raw
native dump is 0.32 s. The fix is decode-once plus a closed-form
rotated-pole index, left for a follow-up.
Fixes two vulnerabilities:
- CVE-2026-48592: Missing authorization on save-job event handler
- CVE-2026-48593: Unbounded cron range expansion causing DoS
- Drop skippy.w5isp.com:8080 as HRRR_BASE_URL in dev/prod config and
all K8s manifests. Default to direct NOAA S3 reads.
- Strip hrrr_cache_dir disk-caching infrastructure from HrrrClient
(read_cache, write_cache, download_and_cache_grib_ranges). Files are
now fetched fresh each time — no local filesystem cache.
- Update stale skippy references in network-policy, test comments, docs.
Microwaveprop.Propagation.Model only exists in lib_ml/ which is excluded
from prod elixirc_paths. Using apply/3 instead of direct @ml_module
calls avoids compile-time warnings when the module is absent. Added
credo:disable-for-next-line inline annotations since Credo flags
apply/3 with known arity but the runtime dispatch is intentional.
Also adds --warnings-as-errors to Dockerfile RUN mix compile so the
build fails on any project-level warning.
Adds a postgres:17-alpine service, a wait-for-postgres step, and a
Run tests step that runs mix ecto.create/migrate/test in the same
docker-run pattern. If tests fail, the Build and push step is
automatically skipped.
The runner is containerised so the host Docker daemon cannot see its
filesystem. Replace `-v ${{ github.workspace }}:/app` with a
`tar | docker run -i ... tar xf -` pipeline to stream the workspace
into the verification container.
P0 (security-critical):
- Gate CSV/ADIF upload tabs behind authentication, add 30s cooldown to all upload handlers
- Cap CSV/ADIF imports at 2,000 rows server-side in both parsers
- Add submitter_verified boolean to contacts (client-cannot-set, anonymous=false)
- Create k8s/secret.example.yaml with placeholders, add LIVE_VIEW_SIGNING_SALT
P1 (high-priority):
- Add Mox.verify_on_exit!() to valkey_test.exs
- Replace DateTime.utc_now() truncation with static ~U literals in map_live_test.exs
- Replace Process.sleep with render_async in pskr_spots_live_test.exs (6 occurrences)
- Add MonitorLive.Show test coverage (4 tests: owner view, non-owner redirect, config success/error)
- Extract duct-detection and mechanism-classification logic from ContactLive.Show into Propagation.PathAnalysis
- Split ContactLive.Show render into 12 function components
- Update CLAUDE.md: remove stale ML model, mark HRDPS active, add backtest/pskr dirs
- Batch CSV import enrichment jobs via new enqueue_for_contacts/1
P2 (medium-priority):
- Set secure:true on session and remember-me cookies in production
- Change SMTP TLS from verify_none to verify_peer with public_key cacerts
- Make /metrics fail-closed in production when PROMETHEUS_AUTH_TOKEN unset
- Add RateLimiter (anon_limit:10, auth_limit:60) to /api/contacts/map
- Add content-security-policy-report-only header
- Add comment noting String.to_atom is compile-time safe in hrdps_client.ex
- Delegate duplicated haversine_km to canonical Microwaveprop.Geo.haversine_km/4
- Consolidate score-tier/color/verdict formatting into Microwaveprop.Format
- Update CLAUDE.md testing section to match actual raw-string-matching practice
- Batch HrrrPointEnqueuer Repo.insert_all calls to single round-trip
- Split weather.ex (1696→216 lines) and radio.ex (1285→54 lines) into purpose-based sub-facades
P3 (low-priority):
- Add LIVE_VIEW_SIGNING_SALT warning comment, extend filter_parameters
- Add host/community validation to snmp_client.ex
- Add raw/1 safety comment in algo_live.ex
- Add hex-audit and cargo-audit Makefile targets
- Add privacy_live smoke test
- Replace notify_listener busy-poll loop with Process.monitor/1 + assert_receive
- Add ContactCommonVolumeRadar changeset validation tests (5 tests)
1. RoverPathProfileWorker sandbox ownership: fallback_hits/2 used
Task.async_stream spawning separate DB-querying processes that
lacked Ecto sandbox ownership in test mode. Replaced with
sequential Enum.map since miss list is ≤9 points — no meaningful
perf impact and eliminates the sandbox race entirely.
2. PSKR client test: asserted '6m' band in defaults, but the actual
microwave band name is '6cm'. Fixed assertion.
Uncommitted work from previous session:
- Add list_users_select/0 helper for admin select dropdowns
- Add regenerate_token/1 context function
- Preload assigned_by association on monitor queries
- Add admin nav link to beacon monitors
- Fix JS event targets (remove redundant target: @myself)
- Add reassign user form to admin monitor show page
- Add regenerate token button to admin monitor show page
- Show assigned_by admin on monitor detail page
- Add beacon monitors section to user management edit page
- Add unassign monitor capability from user management
- Add full test coverage for all new functionality
- Fix pre-existing /account route warning (redirect → /users/settings)
The LiveTable on /admin/contact-edits used the bare ContactEdit schema
as its data source, which caused three symptoms:
- '0 pending' counter but stale approved/rejected edits still visible
- Blank contact/submitted-by cells (select_columns stripped preloaded
associations, cell renderers received flat maps with no :contact/:user)
- Approve/reject didn't remove the row from the table
Fix: assign {Radio, :pending_edits_query, []} as the data_provider in
mount so handle_params threads it to stream_resources. The query variant
of list_resources preserves preloaded associations and includes the
WHERE status = :pending filter.
Added two tests that verify the table rendering and edit removal.
- Add hardware/config fields migration to beacon_monitors table
- Update BeaconMonitor schema with provision/config changesets
- Add context functions: create_hardware, update_config, list_all_monitors
- Remove user-facing monitor creation (browser POST + API POST)
- Update settings page: show assigned monitors table with hardware info
- Update profile page: show assigned monitors, remove register links
- Fix all tests to match new API
Add a Beacon monitors card to /u/:callsign that appears only when the
viewer is the profile owner. Shows registered monitors with truncated
token and last-seen timestamp, with a link to manage monitors in
settings. Includes tests for visibility rules.
Consolidate verify_magic_link/confirm/password_reset_token_query into
one-line delegations to a private helper, removing 29 lines of
duplicated decode/hash/query logic.
- Move ecto.create/ecto.migrate from test alias to test.setup
- Use cmd MIX_ENV=test in precommit to isolate each step in a fresh process
- Move Mox.defmock from test_helper.exs to valkey_test.exs
- accounts.ex: Repo.transaction in ecto_sql 3.14 wraps all return values in {:ok, ...},
causing callers to receive {:ok, {:error, ...}} and {:ok, {:ok, ...}} instead of
direct tuples. Unwrap in update_user_email and update_user_and_delete_all_tokens.
- rover_planning_test.exs: switch to async: false + Oban.Testing manual mode so
RoverPathProfileWorker jobs don't deadlock on sandbox connections inside
Repo.transaction. Introduced create_and_complete_mission!/2 and run_backfill!/1
helpers that separate transaction lifecycle from worker execution.
- Extract shared extract_bearer/1 from api/monitor_auth.ex → api/auth.ex
- Refactor admin_changeset to pipeline through validate_callsign/validate_name/validate_email
- Add limit: 500 to list_beacons/0 and candidate_rover_locations/1
- Combine get_mission_with_paths preloads into single round-trip
The cargo clippy and cargo fmt hooks ran from the repo root but the
Rust project lives in rust/prop_grid_rs/. Add --manifest-path so they
find the Cargo.toml and actually run.
Python pskr_mqtt_listen.py:
- Guard against malformed CONNACK/SUBACK/PUBLISH packets
- Fix _s() truthiness: is not None instead of if v (zero SNR was lost)
- EINTR-safe select loop, OOB data handling, VBInt overflow check
- Proper socket cleanup with try/finally + DISCONNECT on shutdown
Elixir:
- Log dropped spot errors in aggregator (was silently discarded)
- Wire retain_scores_window into NotifyListener chain completion
- Recurse sweep_tmp_dir into band/weather_scalars subdirectories
- Rescue recalibrator.run/0 to write failed status row on crash
- Handle nil in fmt_snr/fmt_callsigns (no more nil dB crashes)
- Replace Stream.run with Enum.reduce logging exits in poll_worker
- Handle File.stat race in ms_footprints prune, grid_center nil log
- Fix unused variables, stale comments, missing get_path!/1
Rust:
- Graceful JoinError handling in fetcher/hrdps_fetcher (no more panics)
- round_to_5min returns Option (leap-second safe)
- Acquire/Release ordering on shutdown flags (was Relaxed)
- Parameterized NOTIFY in db.rs, defensive scalar sweep, NaN guard
- OsString::push for tmp naming, clippy fixes
