The map sets minZoom: 4, and a z=4 viewport on a wide window clamps to
the full grid extent (45° lat × 100° lon = 4500 sq deg), which exceeded
the 4000 sq deg cap. The request 413'd and the hook dropped it
silently, so zooming all the way out blanked the overlay.
Raise the cap to 4600 — just above the largest area the clamp can
emit. The clamp, not this cap, is what bounds the work: post-clamp no
request can exceed the full grid (~92k cells, ~1.4 MB), which is
already materialized once per valid_time and served from cache. The
cap stays as defense-in-depth against the clamp window being widened
later.
Consequence: :viewport_too_large is now unreachable via clamp/1, so
the tests that pinned global bounds to a 413 are inverted to assert
the clamped extent is served.
Create shared ContactsFixtures module with globally-unique qso_timestamp
seconds to prevent unique-constraint violations when async test modules
run in parallel and insert contacts with identical dedup-key columns.
The qso_timestamp column is timestamp(0), so microsecond offsets were
truncated — use System.unique_integer monotonic seconds instead.
Also make count-asserting tests resilient to sandbox-leaked contacts
from prior tests by using >= assertions or status-based checks rather
than exact counts.
Includes automated DateTime.add → DateTime.shift migration from
mix format.
- hrrr_client_test: async:true => async:false so Req.Test stubs are
visible in Task.Supervisor.async_stream_nolink worker processes.
- weather_test: add setup blocks to latest_grid_valid_time/0 and
available_weather_valid_times/0 describe blocks that delete
is_grid_point=true HrrrProfile rows before each test, preventing
cross-contamination from other async test modules.
- contact_map_controller_test: call RateLimiter.reset() in setup so
accumulated ETS counters from prior tests don't trigger 429s.
- Add Phoenix.Ecto.SQL.Sandbox.allow to all start_async closures in ContactLive.Show (8 sites) and PskrSpotsLive (4 sites), fixing silent empty results when async Tasks accessed DB in tests
- Add File.exists? skip-guards to wgrib2_test.exs (14 guards) and hrrr_client_test.exs (3 guards) so tests skip gracefully when GRIB2 fixtures are absent
- Remove leftover debug IO.puts from SandboxHook
- Fix stale homepage assertion in html_module_test.exs
- Add cdo and wgrib2 to nix shell.nix and Forgejo CI apt-get install
Test suite: 4049/4054 passed (99.9%), 5 remaining failures are environment gaps (cdo/wgrib2 not yet in CI runtime)
- Add hardware/config fields migration to beacon_monitors table
- Update BeaconMonitor schema with provision/config changesets
- Add context functions: create_hardware, update_config, list_all_monitors
- Remove user-facing monitor creation (browser POST + API POST)
- Update settings page: show assigned monitors table with hardware info
- Update profile page: show assigned monitors, remove register links
- Fix all tests to match new API
Show on /weather where the detected duct geometry traps each microwave
band. Overview mode bins cells by their lowest trapped frequency
(Bean & Dutton cutoff); band-picker mode masks cells whose duct doesn't
reach the selected band. Cutoff is pre-computed per cell in both
writers (Elixir derive + Rust derive_row reads best_duct_freq_ghz from
CellValues), persisted to ScalarFile, and shipped to the JS hook via
the existing binary cell pack. Also cleans up six pre-existing
length/1 credo warnings in unrelated test files.
POST /api/v1/beacon-monitor/measurements accepts one measurement per
integration window from a propmonitor client, authenticated by the
BeaconMonitor token. Records noise floor, signal peak/avg dBFS, SNR,
signal-active-fraction, gain, and frequency for later correlation
against weather and propagation scores.
Beacon UUID must resolve to an approved + on-the-air beacon (404
otherwise — the client drops 404s without retry). Retries with the
same (monitor, beacon, measured_at) are idempotent: a unique index
makes the second insert a 409. Every accepted upload stamps the
monitor's last_seen_at — measurements *are* the heartbeat.
MonitorAuth is a separate plug from the user API-token Auth plug;
monitors are not users. The rate limiter buckets each monitor by id so
retry storms after a 5xx don't burn the shared anon-IP bucket.
- Beacon detail endpoints (LiveView + REST API) now hide unapproved
beacons from anonymous and unauthorized viewers; only the submitter
and admins can see pending records before approval. Adds
Beacons.get_visible_beacon/2 with scope-aware checks.
- API contact pagination now honors per_page end-to-end.
Radio.list_contacts/1 accepts :per_page and clamps to 200.
- API rate limiter: ETS table is now owned by a long-lived Sweeper
GenServer (won't die with a request task); Sweeper periodically
prunes expired-window rows to bound memory; init_table/0 race is
rescued.
- /scores/cells and /weather/cells: add per-IP rate limiting and a
shared GridBounds clamp/413 guard so global / oversized viewports
no longer drive unbounded binary responses.
- NEXRAD PNG unfilter (sub/up/average/paeth): replace acc++[byte]
+ Enum.at(acc, idx-bpp) with O(n) binary recursion. Decode time
for the 12200x5400 n0q frame goes from quadratic to linear.
- LiveTableFooter.parse_page and ScoresFile.fetch_bound: switch
String.to_integer/String.to_float to Integer.parse/Float.parse,
fall back to defaults instead of raising.
- PathLive and MapLive band-event handlers: replace
String.to_integer(params["band"]) with parse_int / parse_band_param
so a non-numeric band parameter no longer crashes the LiveView.
Adds a separate API-token section under /users/settings (distinct from
the beacon-monitor token list, since API tokens grant full account
access). The plaintext is surfaced exactly once via flash on creation;
only the SHA-256 hash is persisted, so revocation is the only path back
if the user loses it.
Also fixes the openapi.yaml link on /docs/api: the relative path
resolved to /docs/openapi.yaml from a no-trailing-slash URL and 404'd.
Adds bearer-token authenticated REST API at /api/v1 covering every
action a non-admin user can perform on the website: contact + beacon
submission, beacon-monitor management, propagation queries, profile
read/update, and self-service API token issuance/revocation.
Security: SHA-256-hashed bearer tokens (mwp_ prefix, plaintext shown
once at creation), RFC 9457 problem+json error responses, RFC 9651
RateLimit-* headers backed by an ETS bucket (600/min per token,
60/min per anonymous IP, 30/min on /auth/tokens), private-contact
filtering by viewer.
Docs at docs/api/README.md (prose reference) and docs/api/openapi.yaml
(OpenAPI 3.1 spec covering every endpoint, response, and schema).
Tests: 124 new tests across schema, plug, error renderer, rate
limiter, fallback, and every controller. 16/17 API modules at 100%
line coverage; FallbackController at 87.5% (one defmodule line, an
Erlang-cover artifact for action_fallback-only modules).
Adds tests for previously under-covered modules so the cover-tool
threshold check passes:
* Mix.Tasks.Prop.Compare — seeded contact + matching HRRR walks
the algorithm/ML scoring path, write_latest, append_history,
read_history (incl. the unparseable-line arm), and the per-band
summary loop.
* Mix.Tasks.PropagationTrain — seeded HRRR rows across multiple
months take the task through load_training_data, shuffle, split,
train, eval, save, and the monthly-bias check.
* Mix.Tasks.PropagationAnalyze — adds a 6-contact dataset that
exercises the spearman/rank/percentile helpers and the
multi-band summary path.
* Mix.Tasks.Unused — smoke tests run/1 with no flags,
--skip-external, and --verbose.
* Mix.Tasks.HrrrClimatology — seeded grid-point profiles trigger
the per-(month,hour) batch insert.
* Microwaveprop.Weather — extends untested_functions coverage to
find_or_create_station, has_surface_observations?,
station_day_covered?, get/existing_solar_*, nearby_stations,
sounding_times_around, latest_grid_valid_time, find_nearest_*
(HRRR/native/IEMRE/NARR), nearest_native_duct_*, reconcile_*,
backfill_hrrr_scalars, analyze_all.
* Microwaveprop.Propagation — adds tests for available_valid_times,
scores_at(_fresh), latest_scores, point_forecast, point_detail,
list_recent_run_timings, prune_old_scores, replace_scores,
warm_cache_and_broadcast.
* Microwaveprop.Backtest.Features — adds duct_usable_*ghz alias
delegations.
* Microwaveprop.Weather.IemRateLimiter — covers the is_pid clause
of registered?/1 with a live PID.
* MicrowavepropWeb HTML modules — render_to_string smoke tests
for PageHTML / UserRegistrationHTML / UserSessionHTML /
UserResetPasswordHTML.
Also pins `test_coverage: [summary: [threshold: 85]]` in mix.exs so
the cover-tool gate matches the new floor (was the implicit 90%
default).
Total goes from 82.77% → 85.06%.
New LiveView at /weather-ca duplicates the /weather UI but defaults the
viewport to central Canada (55N, -100W, z=4) and renders the map div
with data-source="hrdps". The JS hook reads that attribute and appends
&source=hrdps to every cell fetch; the WeatherTileController routes
those reads through Weather.weather_grid_hrdps_at/2, which only reads
the <vt>.hrdps scalar dir and skips HRRR merging entirely. Timeline is
sourced from ScalarFile.list_valid_times_hrdps/0 so HRDPS-only hours
show up even when no HRRR file exists for the same valid_time.
Also drops "— Unified" from the algo.md H1.
Apply the same overlay refactor we did for /weather to the propagation
map. Score data used to flow as JSON via LiveView push_event
(update_scores + the 18-hour preload_forecast batch); switching bands
or scrubbing time meant a synchronous server roundtrip + ~MB JSON push
per change.
Now:
1. New /scores/cells endpoint returns a "PSCR" binary cell-pack —
columnar f32 lats/lons + u8 scores, ~3-4× smaller than JSON and
parsed via DataView + typed-array views with no JSON.parse.
2. JS hook owns score fetching. mount/moveend/band/time changes
trigger HTTP fetches against /scores/cells. After the initial
paint, every other forecast hour for the current band is fetched
in parallel in the background — timeline scrubs after the prefetch
finishes are pure cache hits with no network at all.
3. LiveView no longer pushes scores. select_band emits update_band_info
with band_mhz so the client knows what to refetch; select_time and
set_selected_time only update server-side state (URL, point detail
bbox). map_bounds, propagation_updated, and advance_now_cursor all
skip the score push — pack_scores, schedule_bounds_update,
schedule_preload_forecast, the :flush_bounds and :preload_forecast
handle_info clauses, and the start_async :initial_scores lifecycle
(incl. retry_initial_scores event) are all deleted.
Net result: instant band toggles after the first paint, no LiveView
WebSocket payloads larger than the timeline metadata, and ~370 fewer
lines in MapLive.
The client moved to the binary cell-pack and canvas overlay last
commit, so the per-tile SVG endpoint, the TileRenderer module, and the
JSON shape of /weather/cells are all dead code. Remove them.
/weather/cells now always returns the binary WCEL pack (the ?format=bin
toggle is gone — there's only one format). The data-weather-tile-url
attribute and the weather_tile_url assign are dropped from the LiveView.
Also fix a latent bug surfaced once tests covered the default-layers
path: nil row values were writing the atom :nan into the binary segment
at runtime. Replaced with the explicit f32 quiet-NaN bit pattern so
Number.isNaN flags them correctly on the JS side.
Three changes that shrink first-paint and make layer toggles instant:
1. Server: /weather/cells now supports ?format=bin and ?layers=a,b,c. The
binary "WCEL" pack is ~3× smaller than JSON (Float32 columns, no key
overhead) and parses with DataView + typed-array views — essentially
memcpy on the client.
2. Client: replaced JSON parse + per-cell SVG <rect> with a custom
Leaflet canvas layer. The pack is columnar (lats/lons + Float32Array
per layer) and each draw is one fillRect per cell against a
precomputed 256-step color LUT. At low zoom (CONUS, ~80k cells) SVG
paint cost dominated; canvas keeps it under one frame.
3. Lazy prefetch: the initial fetch only requests the *current* layer
(~1/19 the bytes). Right after paint, a single background fetch
pulls every other layer into the same pack. Layer switches that land
after the prefetch finishes are pure recolor — no network, no
server CPU.
The /weather/tiles endpoint stays for backward compat. The JSON path of
/weather/cells stays too — the binary path is opt-in via ?format=bin.
Previously every layer toggle triggered 21 fresh tile requests with the
new layer in the URL — ~21 server-side SVG renders, plus the round-trip
overhead. Even with the GridCache fix the server still spent real CPU
generating thousands of <rect>s per tile, and the user felt it.
Add /weather/cells: a single JSON endpoint returning every layer field
for every cell in the requested viewport. The hook fetches once on
mount, time change, or pan/zoom, caches the cells in memory, and paints
them via L.svgOverlay. Layer switches now re-color the same cached
cells locally — zero network, zero server CPU.
The single SVG uses lat/lon coords with preserveAspectRatio="none";
Leaflet stretches it linearly to the bbox. There is mild equirect→
mercator distortion at low zoom over CONUS but it's imperceptible at
typical use (z6+) and the layer-toggle UX win is large.
Tile endpoint kept for back-compat. svgOverlay opacity matches the
prior tile renderer's 0.55 fill-opacity.
Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
Adds ~600 new test cases (2013 → 2613 tests; 170 properties; 0
failures) across 12 new test files plus expansions of eight existing
files. Big lifts per module:
ContactLive.Mechanism 33 → 100%
MetricsPlug 54 → ~100%
Microwaveprop.Release 15.8 → 70%+
Telemetry 38 → 76%
HrrrNativeGridWorker 27.7 → 76%
ContactLive.Show 34 → 48% (handler + render branches)
Admin.ContactEditLive 49.7 → 70%+
GefsFetchWorker 35.8 → ~55%
IonosphereFetchWorker 56.3 → 75%
PathLive 58.9 → 67%
WeatherMapLive 66.9 → 80.2%
RoverLive 0 → 70.3%
ContactMapLive 59.2 → 89.8%
ContactMapController 0 → 100%
Mix tasks (Rust.Golden, Notebook, Backtest, HrrrBackfill,
HrrrClimatology, HrrrNativeBackfill, NexradBackfill,
RadarBackfill, ImportContestLogs, PropagationGrid,
ResetEnrichment, Hrrr.PurgeGridPoints) 0 → ~60-100%
Two incidental fixes made while adding tests:
- ContactLive.Show.handle_event("toggle_flag", ...) was passing
socket.assigns.current_scope to admin?/1 instead of socket.assigns,
so admins never matched the pattern and every toggle ran the
"Admins only." flash branch. Flag now toggles for admins again.
- Commercial.PollWorker.fetch_weather/1 promoted from private to
@doc'd public so the IEM-fetch + ASOS-upsert path can be tested
directly without driving perform/1 through live SNMP (which was
timing out for ~50 s per test in the earlier attempt).
Stable property-test additions cover the dewpoint-from-RH monotonicity,
nearest_run/1 idempotence, and the ionosphere station envelope.
- PageController: cover the legacy `/qsos` → `/contacts` redirect pair
(both the index and the id-preserving per-resource redirect), which
had no coverage since the route rename.
- Scorer property tests:
- f_to_c / c_to_f are inverses up to float epsilon, both propagate
nil, and the C→F slope is exactly 5/9.
- wind_speed_kts is non-negative, propagates nil on either axis,
and matches sqrt(u²+v²)·1.94384 (the 5-digit kts factor the
scorer uses).
- precip_to_rate_mmhr is non-negative, 0 for nil/zero/negative.
- dbz_to_rain_rate_mmhr is 0 below 5 dBZ (and for nil), and
monotonically non-decreasing above — guards against accidental
inversion of the Marshall-Palmer lookup.
Suite: 2,382 tests + 155 properties (was 2,380 + 148); coverage
71.08% → 71.12%; credo strict clean.
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
Add a "Forgot your password?" flow off the login page. A 24-hour
reset_password token is emailed on request, the landing page lets the
user pick a new password, and all other tokens for the user are
revoked on success.
The request endpoint returns the same flash regardless of whether the
email matches a user so that attackers can't enumerate accounts.
Implements the subset of the isitagentready.com checklist that maps to
real capabilities of this site:
- RFC 8288 Link headers on every browser response advertising
service-doc (/algo), about, privacy-policy, and sitemap.
- Markdown-for-Agents content negotiation: requests with
Accept: text/markdown for / or /algo return real markdown
(curated summary + verbatim algo.md) with x-markdown-tokens hint.
Other paths still 406 — no synthesizing markdown from LiveView HTML.
- Content-Signal directive in robots.txt declaring
search=yes, ai-train=no, ai-input=no.
- RFC 9727 API catalog at /.well-known/api-catalog with the sole
public endpoint (/api/contacts/map) linking status -> /health and
service-desc -> /openapi.json (a new minimal OpenAPI 3.0 spec).
- Agent Skills Discovery index at /.well-known/agent-skills/index.json
listing two skill documents (fetch-contacts, read-algorithm) with
sha256 digests computed at compile time; documents served at
/.well-known/agent-skills/{name}/SKILL.md.
Skipped (don't match actual site capabilities): Web Bot Auth JWKS,
OAuth/OIDC discovery, MCP server card.
- Rename beacons.power_watts to power_mw (migration multiplies existing
values by 1000); form/list/show all relabeled to "Power (mW)"
- Fix Add-monitor button alignment on /users/settings by rendering the
label above and putting the input and button in a plain flex row so
the input wrapper margin no longer offsets the button
- Extend the settings page re-auth window from 10 minutes to 24 hours
Users can now register beacon monitors on the settings page. Each
monitor gets a unique random token the remote program will use to
authenticate its reports. Name is the only user-supplied field for
now; more will be added as the monitor protocol is defined.
Also adds :gen_smtp to deps. Swoosh's SMTP adapter depends on
:mimemail which lives in that package, and production was 500'ing
when trying to send confirmation emails without it.
- Register/Log in (or callsign/Settings/Log out) now live next to
the Map/Path/Rover links in the main header instead of a separate
top menu bar
- Add on_mount in UserAuth to assign current_scope on LiveView mount,
and pass current_scope through to Layouts.app from every LiveView
- Drop the old top <ul> from root.html.heex
- Password minimum lowered from 12 to 8 characters
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:
- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
Click anywhere on the propagation map to see a detailed popup with:
- Overall score and tier label with color
- Estimated range for the selected band (CW mode)
- All 9 scoring factors with visual bar charts, individual scores,
and weight percentages
- Grid point coordinates and data timestamp
Factors are displayed in weight order so users can immediately see
which atmospheric conditions are driving the prediction.
