Raise local_limit 2→4 (12 concurrent cluster-wide) and rate_limit 10→30
per hour. Each job makes 2 serial CDS requests, so active workers map 1:1
to CDS slots; 12 sits safely under CDS-Beta's ~16 per-user ceiling while
the rate gate stays above steady-state so it stops being the bottleneck.
Contact detail page now renders an atmospheric profile section backed by
HRRR when available and ERA5 (reanalysis) otherwise, labeled with the
data source. When both sources are missing and HRRR is known-unavailable
(pre-2014 contacts), the Era5FetchWorker is enqueued so the month-tile
backfill gets triggered from a user visit.
Status page ERA5 progress previously computed "complete" as
(total_contacts - hrrr_unavailable_contacts), which treated every contact
with any HRRR status as ERA5-complete — inflating the bar to 76% while
era5_profiles had 0 rows. Replaced with a per-candidate EXISTS query
against era5_profiles using the same 0.15°/30-min tolerance
Weather.find_nearest_era5/3 uses at lookup time.
CDS rejects `reanalysis-era5-pressure-levels` requests that include
dewpoint_temperature because it's only a single-level 2m variable; no
pressure-level dewpoint exists in ERA5. The failure mode is a 2–3 second
`{"status": "failed"}` response with no message field, which is why
prod was burning through Era5MonthBatchWorker attempts without anything
landing in era5_profiles.
Switch the pressure-level moisture variable to specific_humidity (SPFH),
update the wgrib2 extraction regex, and convert SPFH → dewpoint °C via
Weather.ThetaE.dewpoint_from_spfh/2 in the profile builder. Surface
dewpoint keeps using the single-level 2m dewpoint_temperature, which is
valid and already working.
Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
- Tailwind source(none) wasn't scanning deps, so shadcn tokens
(border-border, bg-muted, text-foreground, ...) used by live_table
and sutra_ui weren't generated, leaving `border` to fall back to
currentcolor — a heavy dark border around every table in light mode.
Add @source directives for deps/live_table/lib and deps/sutra_ui/lib.
- Drop the hardcoded width:100% on the outer .select wrapper so the
per-page selector's Tailwind w-24 wins instead of stretching across
the header and overlapping the search box.
- Make whole rows clickable: global click delegator in app.ts forwards
tr clicks to the first <a href> in the row (the Actions "View" link),
skipping inner interactive elements. cursor + hover highlight added.
- Replace the default Prev/Next footer with a daisyUI .join/.btn-based
pager (MicrowavepropWeb.LiveTableFooter), wired up globally via
:live_table defaults so it applies to every live_table at once.
warm_grid_cache_and_broadcast was re-SELECTing 92k hrrr_profiles rows
with JSONB profile/duct_characteristics columns on every forecast hour.
Row decoding exceeded the 15s default DB connection timeout, killing
the worker before f01-f18 could run. Oban retried from f00 and got
stuck in a loop: for the last several hours, no forecast hours were
being written and even f00 was stale.
Build the weather cache rows directly from the in-memory grid_data
the worker already has (Weather.build_grid_cache_rows/2) and
GridCache.broadcast_put directly. No DB round trip, no JSONB decode.
Also widen the cron from hourly to every 3 hours so a full f00-f18
sweep (~95 min) can actually complete before the next run starts,
and drop the redundant prune_old_scores() at worker start
(PropagationPruneWorker already runs every 15 min). Add a 60s query
timeout to load_weather_grid_from_db as defense-in-depth for the
cold-cache fill path that still uses it.
- Compat CSS layer (assets/css/live_table_compat.css) maps the
shadcn tokens live_table/sutra_ui use (bg-muted, text-foreground,
border-border, bg-background, select-trigger, input-group, etc.)
onto daisyUI base colors, and provides hand-written component CSS
for select/dropdown/input-group/empty so the table has a proper
surface + a select popover that actually hides when closed.
- Default-sort the Contacts table by most recent inserted_at.
- Beacon detail "Plot path" now encodes lat/lon to an 8-char
Maidenhead grid when coordinates are known, so the destination
handed to the path calculator is more precise than the beacon's
stored 4-char grid.
- Rename BackfillLive -> StatusLive, move route from /admin/backfill
to /status, retitle "Backfill Dashboard" -> "Status", drop the
enqueue form (LiveStash + BackfillEnqueueWorker no longer needed
on this page). Contact edit admin back-link now points at /status.
TerrainAnalysis.deygout_diffraction/2 short-circuits to integer 0
when the path has no interior points or the principal edge nu is
below the shadow boundary, which then made Float.round/2 blow up
inside compute_loss_budget when the rest of the losses were all
floats. Multiply the diffraction value by 1.0 before it feeds into
the total so clear paths coerce cleanly without touching
TerrainAnalysis's (number()) contract.
Replace the stacked-bar 18-hour forecast on the path calculator with
an SVG line chart that mirrors the look of the point-detail sparkline
on /map:
- polyline through every hour with per-tier stroke color
- score gridlines at 0 / 50 / 100
- white "now" marker outlined in the tier color
- highlighted best-hour dot with score label
- first / middle / last time labels along the x-axis
- Improving / Steady / Declining trend chip in the header
Rendered server-side in HEEx as a viewBox="0 0 300 96" SVG so it
scales responsively and inherits the daisyUI text color for axes.
The Best / Worst summary line under the chart is preserved.
Also fixes the Path Weather Avg card: the "Refrac Gradient" cell was
the only stat that split its units out onto a second line in a
smaller muted row. Move "N/km" up beside the value so the cell reads
the same as Pressure (mb), PWAT (mm), BL Depth (m), etc.
HEEx attribute values aren't run through Elixir's string escape
machinery, so "Feed Loss (\u00d72)" was showing up verbatim in the
Power Budget card instead of collapsing to the multiplication sign.
Swap the escape for the literal U+00D7 character.
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.
New modules:
- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
Looks up from qrz_callsigns first, falls back to a live fetch, and
upserts the result with a configurable cache_ttl_hours (default
168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
https://xmldata.qrz.com/xml/current/. Holds the session key in an
Agent, transparently re-logs-in on :session_expired, and parses
responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
we actually consume (identity, name, grid, address, lat/lon).
The full XML payload stays in the raw :data jsonb column for
anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
Geocoding API. Only called as a fallback when QRZ has no explicit
<lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
callsign_locations cache, on miss calls Qrz then either uses QRZ's
coords directly or geocodes the formatted address, snaps to an
8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
upserts the result.
Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.
Wiring:
- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
creates qrz_callsigns and callsign_locations with unique indexes
on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
never short-circuits test-level stubs, and supplies dummy QRZ
credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
and GOOGLE_API_KEY from the environment so prod and dev can
configure both upstream keys out of band.
Tests (ported verbatim from gridmap-web):
- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
behavior, upsert on stale, error passthrough, case-insensitive
input
- test/microwaveprop/geocoder_test.exs — success, zero results,
request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
including the QRZ-lat/lon shortcut and the missing-address error
path
All 1294 tests still pass. Credo strict clean.
Replaces the hand-rolled sort/search/pagination on /contacts with
LiveTable.LiveResource. Sortable columns for station1/2, grid1/2,
band, mode, distance_km, qso_timestamp, and inserted_at; searchable
across both stations, both grids, and mode. Custom renderers preserve
the enrichment summary badge (with per-status tooltip), the invalid
flag column, and the qso/inserted_at formatters.
The reciprocal-grouping block has been dropped per the brainstorming
decision — every QSO now shows as its own row, which avoids fighting
live_table's one-row-per-record model. A "View" action links to the
detail page instead of the previous row-click handler (live_table's
stream dom_ids are random UUIDs so row-click by record id is no
longer practical without forking the library).
Test updates:
- sort test switched to ?sort_params[station1]=asc (live_table URL
format; the old ?sort_by/?sort_order params are gone)
- pair-search test removed since live_table does a single ILIKE
across searchable fields and no longer supports the two-callsign
intersection that Radio.list_contacts/1 used to do; single-term
search is exercised instead
- pagination test asserts the presence of "Page" (live_table's
pagination text is "Page N" without the total count)
Also converts the stray cond-with-single-clause in
UserManagementLive.Index's delete handler to if/else (credo fix).
The pending-edits list on the admin review page now sorts through
live_table. Columns for contact, submitted-by user, changed-fields
summary, and submitted timestamp are rendered via 2-arity custom
renderers so the Ecto-preloaded :contact and :user associations stay
accessible — live_table's data_provider path skips select_columns so
the full struct (with preloads) reaches the renderer.
The review drawer, diff table, approve/reject flow, and note textarea
all stay as-is; after approve/reject the page push_patches back to
its current path so live_table re-runs handle_params and the list
refreshes.
Pending count in the header now reads Radio.pending_edit_count/0
instead of length(@edits) since we no longer hold the full list in
socket state.
Replaces the stream-backed static table for approved beacons with
LiveTable.LiveResource, giving users column sorting and full-text
search across callsign and grid. Custom renderers preserve the
frequency/EIRP formatting, keying label, and on-air badge so the
visual output matches the previous daisyUI cells.
A new Beacons.approved_beacons_query/0 returns the base Ecto query
(`where: approved == true`), assigned onto socket.assigns.data_provider
in mount so live_table's handle_params uses it as the root query and
applies sort/search/paginate on top. The leaflet map above the table
keeps its existing phx-hook+phx-update=ignore setup, and the pending-
approval admin section stays as a stream-backed `<.table>` — live_table
is single-resource-per-liveview and the pending list is a distinct
admin concern anyway.
PubSub updates (beacon created/updated/deleted) now push_patch to the
current path so live_table re-runs handle_params and the table
refreshes in place.
Adds an "Open in Path Calculator" action to the contact detail
header. The link navigates to /path with source/destination prefilled
(preferring Maidenhead grids over station callsigns, since grids
are exact and callsigns go through a QRZ + geocoder round-trip)
and the contact's band selected. PathLive's handle_params already
auto-runs the calculation when source + destination are present,
so the user lands straight on the computed path ready to tweak
TX power, antenna heights, and gains.
TerrainAnalysis.analyse was being called with default 0 m antenna
heights on both ends of a QSO's path, which makes the knife-edge
diffraction math pretend the radios are sitting directly on the
terrain surface. Pass 3.048 m (10 ft) for both ant_ht_a and ant_ht_b,
and shift the forward/reverse elevation angles to match. This is
only a display assumption on /contacts/:id — terrain worker paths
computed elsewhere are unaffected.
Hovering the dotted-underlined "callsign" in the Destination label
on /path now surfaces a tooltip explaining that callsign lookup
queries QRZ for the licensee's mailing address and geocodes it, so
the resulting position may differ from where the station actually
operates. Gives users a reason to enter a grid directly when they
know it's more accurate than a QRZ address.
Wire up gurujada/live_table 0.4.1 as a dependency (alongside its
transitive sutra_ui and optional igniter requirement) with the
:live_table config pointing at our Repo and PubSub. Forced oban_web
override so the path-pinned vendored copy still wins over live_table's
hex constraint.
Convert the admin /users page to use LiveTable.LiveResource with
sortable, searchable columns for callsign, name, and email. Custom
renderers preserve the daisyUI admin badge and the pending/confirmed
date cell. Hidden id field stays in the query so the edit and delete
actions can target a record by id.
Updated the delete test to push the delete event directly with an id
payload; the stream dom_ids under live_table are random UUIDs so
tr#users-<id> selectors no longer work.
Collapse the three admin-only links (Users, Contact edits, Oban) into
a single "Admin" dropdown in the horizontal nav so the top bar isn't
cluttered with admin controls for the one admin account. Non-admins
see nothing extra.
The profile page's involving-contacts query now limits to the 100
most recent rows (ordered by qso_timestamp desc). A small note is
shown under the heading when the cap is hit so users know there's
more data they're not seeing.
Clicking anywhere in a row on the /u/:callsign involving-contacts
table now navigates to that contact's detail page, matching the
click affordance on the main /contacts list.
Add a new card on the profile page listing every contact where the
callsign appears as either station1 or station2, case-insensitive,
newest first. Backed by a new `Radio.list_contacts_involving_callsign/1`
query with unit coverage for station1/station2 matching, case handling,
ordering, and blank input.
Move Scoring Algorithm directly above About in map and weather-map
sidebars; reorder the horizontal nav so Submit sits before Contacts
to match.
Add a "Plot path to beacon" button on the beacon detail page that
navigates to /path with destination, band (snapped via BandResolver),
and dst_height_ft prefilled from the beacon.
/beacons gains a Leaflet map above the table showing every approved
beacon as a circle marker with a popup that links to the detail page.
New BeaconsListMap JS hook (assets/js/beacons_list_map_hook.ts) walks a
data-beacons JSON payload, fitBounds over the marker set, and mirrors
the format_freq/1 integer-comma formatting used on the detail view so
the popups match the rest of the UI. Approved/on-air beacons render
green, off-air beacons render gray. BeaconLive.Index.mount/3 now
materializes the list into an assign so it can encode it into the data
attribute; the live PubSub path keeps the map in sync on
create/update/delete.
Profile (/u/:callsign) polish: drop the hero-radio, hero-signal, and
hero-information-circle icons per request, and replace the daisyUI
`avatar placeholder` wrapper with a plain flex-centered circle so the
initial letter actually lives inside the circle instead of anchored to
the top-left corner (daisyUI 5 removed the old placeholder-centering
behavior).
Navigation ordering: in the top navbar (layouts.ex) Contacts now sits
immediately before Contact Map instead of after it. All three vertical
sidebars already had them adjacent in the correct order, so only the
horizontal nav changed.
Profile page:
* New MicrowavepropWeb.UserProfileLive at /u/:callsign is a public
page showing a user's contacts and beacons. Resolves case-
insensitively so /u/w5isp and /u/W5ISP are the same thing; unknown
callsigns redirect to /. Uses daisyUI card / stats / table
components with an avatar-placeholder initial and hero icons.
* Accounts.get_user_by_callsign/1 (case-insensitive) plus
Radio.list_contacts_for_user/1 and Beacons.list_beacons_for_user/1
back the page. Beacons list includes both approved and pending so
owners see their drafts.
* The top nav bar and the three LiveView sidebars (MapLive,
WeatherMapLive, ContactMapLive) now render the logged-in callsign
as a navigate link to /u/:callsign instead of a static label.
* Nine new tests cover the lookup, the LiveView render, and the
ownership-scoped queries.
Flexible band input:
* New Microwaveprop.Radio.BandResolver module converts any of:
ADIF wavelength labels ("33cm", "1.25cm", "6mm", case/whitespace
insensitive), numeric frequency strings ("903.100", "10368.000"),
and canonical MHz integers into the one of the site's known bands.
Returns the nearest allowed band for numeric inputs >= 900 MHz,
nil otherwise.
* 902 MHz is added to Contact.@allowed_bands, ContactEdit.@allowed_bands,
AdifImport.@allowed_bands, and the BandResolver list so "33cm"
round-trips end-to-end.
* AdifImport and CsvImport now delegate band resolution to
BandResolver, and Radio.create_contact/2 normalizes the :band attr
on the way in so the manual form and any API callers benefit too.
CsvImport's "invalid band" tests previously used 99999 MHz which
the new resolver snaps to the nearest allowed band; swapped to
"notaband" which is truly unresolvable.
Contacts and beacons list UX:
* Remove the "Submitted" column from /contacts — it duplicated info
already visible on the detail page and was pushing the real
columns off narrow viewports. submitted_cell/1 and its three
column-specific tests go with it.
* Hide the Lat / Lon columns from /beacons — six decimal places of
coordinates weren't useful next to the grid square and took a
disproportionate amount of row width.
Two bugs on the propagation map:
1. LiveStash reconnect path crashed the LiveView with KeyError on
:initial_scores_json. stash_assigns/2 only persists selected_band and
selected_time, but the recovered-socket branch returned early without
rebuilding any of the other mount-time assigns (bands, bounds, the
initial score JSON payload, grid/radar toggles, antenna height).
Users on the Phoenix longpoll fallback — or anyone whose websocket
reconnected after a brief network blip — saw a blank page and a
server-side 500. Refactor mount/3 to always compute ephemeral assigns,
using recovered selected_band / selected_time when present and
falling back to defaults otherwise.
2. The propagation reach polygon shown after clicking a point never
redrew when the user changed bands or scrubbed the forecast timeline.
The redraw logic only ran inside the point_detail handler and was
gated on `hull.length >= 3`, so the stale polygon from the old band
stuck around whenever the new band had no coverage at the clicked
point. Extract a redrawReachPolygon/0 helper on the Leaflet hook that
always clears the previous polygon first and pulls the tier color
from the current score at the clicked location, then call it from
update_scores (band change + server time scrub) and the cached
timeline-scrub path. The point_detail handler delegates to the same
helper so the polygon stays in sync with both the score grid and the
detail panel.
New "Weather radar" toggle in /map's control panel (mobile + desktop)
layers an ECCC GeoMet WMS composite dBZ mosaic over the propagation
heatmap. The Radar_1km_dBZ-Extrapolation layer covers CONUS + Canada at
1 km resolution, updated every ~6 minutes. Tiles are lazy-created on
first toggle so visitors who never open the panel never hit the WMS
service. Refresh is driven by a setInterval that calls setParams with a
cache-busting timestamp, matching ECCC's publish cadence.
Rain-scatter display (NEXRAD cell markers + "Rain Scatter" section in
the point-detail panel) is disabled for now. The live radar overlay
answers "is there rain near this point?" more directly, and the scatter
propagation feature needs more work before it's useful. The Elixir
start_async, handle_async, and fetch_rain_scatter helpers are removed.
The JS drawScatterMarkers / buildScatterBlock paths remain as harmless
dead code since they're guarded by a `rain_scatter` field the server no
longer sends — easy to re-enable later by restoring the payload key.
Submit and contact-detail forms now use a plain text input for the QSO
timestamp with a "YYYY-MM-DD HH:MM" placeholder and regex pattern. The
native datetime-local input ignores lang="en-GB" on macOS/iOS when the
system clock is 12-hour, so it was rendering AM/PM against the user's
wishes. Ecto's :utc_datetime cast already accepts this format (verified
space+seconds, T+seconds, and both with and without seconds/Z).
Contact edit workflow grows a third branch: admins still apply directly,
non-owners still go through the admin review queue, and now logged-in
owners of a contact (user_id == current_user.id) can apply edits
directly too. New helpers Radio.owner?/2 and Radio.apply_owner_edit/3
reuse the existing normalize + diff + apply_edit_to_contact path so the
grid-change → enrichment re-enqueue pipeline kicks in automatically.
Anonymous contacts (user_id nil) and other users' contacts both return
{:error, :not_owner}.
Nine new tests cover owner?/2 and apply_owner_edit/3 including the
no-changes, wrong-user, and anonymous-contact paths.
Regex.scan with `return: :index` reports BYTE offsets, but String.slice/3
uses CHARACTER offsets. When a record contained a multi-byte UTF-8
character in an earlier field (e.g. "café" in NOTES), every subsequent
field's slice shifted one byte left, eventually landing on a literal
">" that crashed String.to_integer/1 with ArgumentError.
Switch to :binary.part/3 which is byte-based and matches what Regex.scan
reports. Also simplify the value offset calculation to use the scanned
tag_len directly instead of re-splitting the tag text. Reproduced in a
new regression test.
Prod stack trace: AdifImport.parse_fields/1 crashed on upload with
binary_to_integer(">") -- a FreeDV/N1MM logger that embedded UTF-8
characters in a NOTES field triggered it.
MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
The contact detail page used to gather every enrichment source in one
parallel_hydrate/2 bundle and await all five tasks before updating any
assigns, so users watched the whole right-hand column flash in at the
speed of the slowest query (usually hrrr_profiles_for_path against a
42M-row table).
Each source now runs in its own start_async task: :weather, :solar,
:hrrr_path, :terrain, :iemre. Matching handle_async/3 clauses assign
the slot and call refresh_computed/1, which rebuilds elevation_profile
+ propagation_analysis + data_sources from whatever's currently in
assigns. Sections appear independently as their data arrives, and the
existing PubSub handlers (:hrrr_ready, :weather_ready, :terrain_ready,
:solar_ready) still pick up later backfill completions the same way
they did before.
Centralized the From and Reply-To headers in Microwaveprop.Mailer.apply_defaults/1
so UserNotifier and EditNotifier can't drift. Drops the EMAIL_FROM env var
override — both values are hardcoded now.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
Right-hand dark sidebar holds the layer selector, description,
navigation links, and auth menu. Mobile floating panel mirrors the
pattern used on /map so users get the same controls in both views.
- Forecast hour clicks showed only a small square of coverage because
:preload_forecast ran at mount using a hardcoded fallback bounding box
instead of the client's actual viewport. Move the preload trigger into
handle_event("map_bounds", ...) so the cache always matches the
viewport the client just asked about.
- Map overlay blinked on load/update because renderScores called Leaflet
GridLayer.redraw(), which removes every tile element before recreating
them. Repaint the existing tile canvases in place instead.
- Prod SMTP to mail.smtp2go.com was failing with an "unexpected_message"
TLS alert: gen_smtp wasn't sending SNI so the wildcard cert endpoint
rejected the handshake. Set tls: :always with explicit tls_options
(server_name_indication, versions, verify).
- AdifImport.normalize_mode/2 maps ADIF MODE (and SUBMODE when present)
onto the fixed set of modes the site accepts: CW/SSB/FM/FT8/FT4/Q65.
SUBMODE takes precedence so `MODE=MFSK, SUBMODE=FT8` correctly imports
as FT8. USB/LSB/DSB all become SSB; NBFM/WFM become FM.
- Unknown modes (RTTY, PSK31, etc.) map to nil so the row still imports
— mode is optional on the submission path now.
- AdifImport.mode_mapping_reference/0 returns the user-facing list of
recognized inputs and their mappings.
- Submit page's ADIF tab renders that list in a table so users can see
upfront what will happen to their file before uploading.
- Add ContactMapController serving /api/contacts/map with pre-gzipped
JSON from Microwaveprop.Cache. Browsers handle Content-Encoding: gzip
natively; payload drops from 5.7 MB raw to 1.5 MB on the wire (73%).
Cache TTL 10 min, invalidated on contact insert alongside the other
contact-map cache keys.
- Move contact map payload out of the LiveView's initial HTML entirely.
Shell renders with the filter chrome (count + band list); the hook
fetches contacts from the HTTP endpoint in parallel with map init.
This also avoids Phoenix.Socket's 64 KB frame limit (push_event would
have required chunked transfer or a global frame size bump).
- Parallelize ContactLive.Show hydrate: five independent DB loads
(weather, solar, hrrr_path, terrain, iemre) now run concurrently via
Task.async + Task.await_many. Total latency drops from sum(each) to
max(each), cutting contact show hydrate time roughly 5x.
- Add Weather.GridCache: ETS cache of derived HRRR grid rows keyed by
valid_time, cluster-synced via PubSub. Eagerly warmed from
PropagationGridWorker after each upsert so /weather map pan/zoom and
weather_point_detail hit zero DB on warm cache.
- Replace latest_weather_grid DB query path with cache-first lookup +
DB fallback. hrrr_profiles is 42M rows partitioned; pulling 3-10k
rows per viewport on every pan was the main cost.
- ContactLive.Show: defer the heavy enrichment loads (weather, solar,
HRRR, terrain, IEMRE, elevation profile, ITU-R propagation analysis,
data_sources) into a handle_info(:hydrate) that runs after the shell
renders. Initial mount now returns nil placeholders; template already
had :if guards for all of them. Shell-to-first-paint goes from
~500ms-2s down to ~20ms.
- Cache fetch_queue_counts for 5s in ContactLive.Show — oban_jobs group
by query was running on every contact page view.
- Backfill stats: wrap count_unprocessed, fetch_stats, fetch_db_stats
in Microwaveprop.Cache with 2-5s TTLs; bump refresh debounce from 1s
to 2s so bulk enrichment events don't thrash the DB.
- Add contacts_inserted_at_desc_id_desc_idx so /contacts list runs as an
Index Scan (~0.4ms) instead of a Seq Scan + top-N heapsort (~77ms on
58k rows)
- Add Microwaveprop.Cache: tiny ETS-backed TTL cache for memoizing
expensive-but-stale-tolerant values
- Cache total_entries for unsearched /contacts page loads (30s TTL,
invalidated on insert)
- Cache the entire /contacts/map payload (pre-encoded JSON + band list),
moving load_contacts into Radio.contact_map_payload; invalidated on
new contact insert. Mount goes from ~150-300ms to ~5ms.
- Fix /map overlay blinking on load: reuse the Leaflet GridLayer across
renderScores calls and just redraw() against the updated ScoreGrid,
instead of removeLayer + new layer which flashed between tile regens
- Make mode optional on submission: schema change (null: true),
submission_changeset no longer requires it, blank strings normalise to
nil, CSV import accepts 6-column (no mode) or 7-column layouts
- core_components .input adds a red asterisk to labels when required,
giving users a visual cue for which fields must be filled on /submit
- ScoreCache stores {band, valid_time} as %{{lat, lon} => score} map so
point lookups are O(1); adds fetch_point/4 and valid_times/1
- available_valid_times/1 reads directly from ScoreCache when warm,
falls back to DB on cold start
- point_forecast/3 iterates cached valid_times and uses fetch_point/4
instead of hitting the DB per click
- NexradCache: node-local ETS cache of decoded n0q PNG pixel buffers
keyed by 5-minute rounded timestamp; skips ~1-5s HTTP+decode on
concurrent/repeat clicks within the same window
- MapLive: start_async the rain_scatter fetch so point_detail renders
immediately with a pending marker; push rain_scatter_update when
NEXRAD resolves
- MapLive: preload all 18 remaining forecast hours for the current
viewport after mount/band change/propagation_updated; client caches
them and renders timeline scrubs instantly without a server roundtrip.
Adds set_selected_time event for fast-path state sync.
- Propagation map JS: forecastCache map + drawScatterMarkers helper,
timeline click uses preloaded cache when available
- Add ScoreCache GenServer with node-local ETS table keyed by
{band, valid_time}, subscribed to "propagation:cache" PubSub topic so
every pod stays in sync with a single hourly compute
- scores_at/3 checks cache first, falls back to DB and populates on miss
- PropagationGridWorker warms and broadcasts the cache for each band
after every forecast hour upsert; prunes >2h old entries
- Replace per-pixel string-keyed Map with flat Int8Array over the CONUS
grid in propagation_map_hook.ts to eliminate allocations in the tile
rasterization hot loop (interpolateScore / propagationReach)
