CDS rejects `reanalysis-era5-pressure-levels` requests that include
dewpoint_temperature because it's only a single-level 2m variable; no
pressure-level dewpoint exists in ERA5. The failure mode is a 2–3 second
`{"status": "failed"}` response with no message field, which is why
prod was burning through Era5MonthBatchWorker attempts without anything
landing in era5_profiles.
Switch the pressure-level moisture variable to specific_humidity (SPFH),
update the wgrib2 extraction regex, and convert SPFH → dewpoint °C via
Weather.ThetaE.dewpoint_from_spfh/2 in the profile builder. Surface
dewpoint keeps using the single-level 2m dewpoint_temperature, which is
valid and already working.
Three new analysis passes in scripts/recalibrate_algo.py that match the
scoring factors landed in commit 86364f4:
* NEXRAD composite reflectivity vs distance per band — checks that the
Marshall-Palmer rain rate in Scorer.dbz_to_rain_rate_mmhr/1 moves the
needle in the expected direction at 24+ GHz.
* hrrr_native_profiles.best_duct_band_ghz vs distance per band —
validates the 1.15× boost in Scorer.score_refractivity/4. Current
corpus shows essentially zero contacts with a native duct supporting
≥5 GHz at the target band, consistent with the 0.12% rate in Part 2c;
the boost is physics-correct but almost never fires today.
* Commercial-link rx_power degradation vs contemporaneous DFW contacts —
pearson + binned distance. Empty today because commercial_samples only
covers Mar 30–Apr 13 and contests don't overlap. Scaffolding is in
place so the moment contest-season overlap exists, the signal shows.
Report rebuilt at docs/algo-reports/2026-04-13-recalibration.md.
Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
daisyUI 5 defines `.select` with its own border, chevron background-image,
padding, width clamp and box-shadow. Those rules were bleeding through
onto sutra_ui's wrapper `<div class="select">` in the live_table per-page
selector, rendering two chevrons and the wrong width. Explicitly zero out
every visible property except `width`, so Tailwind width utilities (w-24)
still win and the inner `.select-trigger` button owns the visible styling.
- Tailwind source(none) wasn't scanning deps, so shadcn tokens
(border-border, bg-muted, text-foreground, ...) used by live_table
and sutra_ui weren't generated, leaving `border` to fall back to
currentcolor — a heavy dark border around every table in light mode.
Add @source directives for deps/live_table/lib and deps/sutra_ui/lib.
- Drop the hardcoded width:100% on the outer .select wrapper so the
per-page selector's Tailwind w-24 wins instead of stretching across
the header and overlapping the search box.
- Make whole rows clickable: global click delegator in app.ts forwards
tr clicks to the first <a href> in the row (the Actions "View" link),
skipping inner interactive elements. cursor + hover highlight added.
- Replace the default Prev/Next footer with a daisyUI .join/.btn-based
pager (MicrowavepropWeb.LiveTableFooter), wired up globally via
:live_table defaults so it applies to every live_table at once.
warm_grid_cache_and_broadcast was re-SELECTing 92k hrrr_profiles rows
with JSONB profile/duct_characteristics columns on every forecast hour.
Row decoding exceeded the 15s default DB connection timeout, killing
the worker before f01-f18 could run. Oban retried from f00 and got
stuck in a loop: for the last several hours, no forecast hours were
being written and even f00 was stale.
Build the weather cache rows directly from the in-memory grid_data
the worker already has (Weather.build_grid_cache_rows/2) and
GridCache.broadcast_put directly. No DB round trip, no JSONB decode.
Also widen the cron from hourly to every 3 hours so a full f00-f18
sweep (~95 min) can actually complete before the next run starts,
and drop the redundant prune_old_scores() at worker start
(PropagationPruneWorker already runs every 15 min). Add a 60s query
timeout to load_weather_grid_from_db as defense-in-depth for the
cold-cache fill path that still uses it.
- Compat CSS layer (assets/css/live_table_compat.css) maps the
shadcn tokens live_table/sutra_ui use (bg-muted, text-foreground,
border-border, bg-background, select-trigger, input-group, etc.)
onto daisyUI base colors, and provides hand-written component CSS
for select/dropdown/input-group/empty so the table has a proper
surface + a select popover that actually hides when closed.
- Default-sort the Contacts table by most recent inserted_at.
- Beacon detail "Plot path" now encodes lat/lon to an 8-char
Maidenhead grid when coordinates are known, so the destination
handed to the path calculator is more precise than the beacon's
stored 4-char grid.
- Rename BackfillLive -> StatusLive, move route from /admin/backfill
to /status, retitle "Backfill Dashboard" -> "Status", drop the
enqueue form (LiveStash + BackfillEnqueueWorker no longer needed
on this page). Contact edit admin back-link now points at /status.
Since da6b312 (Add live_table dep), every main-branch CI build has
silently failed on `mix deps.compile` inside the Docker builder. The
failure is that live_table ships two unguarded mix tasks —
lib/mix/tasks/live_table.install.ex and
lib/mix/tasks/live_table.gen.live.ex — both of which do
`use Igniter.Mix.Task` at the top level without wrapping it in a
`Code.ensure_loaded?(Igniter.Mix.Task)` check. When Igniter isn't
available, compiling the live_table dep blows up with
`module Igniter.Mix.Task is not loaded`.
I had declared igniter as `only: [:dev, :test], runtime: false`
which meant it was only fetched in those envs, so the prod Docker
build never had it and failed there. The blast radius is every
commit from da6b312 through 263c1bb (14 commits) none of which ever
made it to prod:
- /users, /beacons, /contacts, /admin/contact-edits live_table
conversions
- Admin nav dropdown
- Cap "Contacts I'm in" at 100
- Contact detail path-calc link + 10 ft AGL assumption
- Path calculator callsign tooltip
- Drop gridmap.org and do QRZ + Google geocoding locally
- Feed Loss (×) HEEx escape fix
- Path forecast line chart and refractivity gradient unit fix
- Float.round/2 crash fix on clear-terrain diffraction loss
Drop the env restriction on igniter so it's available to the prod
build as well. `runtime: false` keeps it out of the release bundle
so there's no runtime bloat, it's strictly a compile-time build
tool for the live_table dep.
Verified locally with a clean `_build/prod` + `MIX_ENV=prod mix
deps.compile` (all 54 deps including live_table now build) and the
full test suite still passes.
TerrainAnalysis.deygout_diffraction/2 short-circuits to integer 0
when the path has no interior points or the principal edge nu is
below the shadow boundary, which then made Float.round/2 blow up
inside compute_loss_budget when the rest of the losses were all
floats. Multiply the diffraction value by 1.0 before it feeds into
the total so clear paths coerce cleanly without touching
TerrainAnalysis's (number()) contract.
Replace the stacked-bar 18-hour forecast on the path calculator with
an SVG line chart that mirrors the look of the point-detail sparkline
on /map:
- polyline through every hour with per-tier stroke color
- score gridlines at 0 / 50 / 100
- white "now" marker outlined in the tier color
- highlighted best-hour dot with score label
- first / middle / last time labels along the x-axis
- Improving / Steady / Declining trend chip in the header
Rendered server-side in HEEx as a viewBox="0 0 300 96" SVG so it
scales responsively and inherits the daisyUI text color for axes.
The Best / Worst summary line under the chart is preserved.
Also fixes the Path Weather Avg card: the "Refrac Gradient" cell was
the only stat that split its units out onto a second line in a
smaller muted row. Move "N/km" up beside the value so the cell reads
the same as Pressure (mb), PWAT (mm), BL Depth (m), etc.
HEEx attribute values aren't run through Elixir's string escape
machinery, so "Feed Loss (\u00d72)" was showing up verbatim in the
Power Budget card instead of collapsing to the multiplication sign.
Swap the escape for the literal U+00D7 character.
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.
New modules:
- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
Looks up from qrz_callsigns first, falls back to a live fetch, and
upserts the result with a configurable cache_ttl_hours (default
168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
https://xmldata.qrz.com/xml/current/. Holds the session key in an
Agent, transparently re-logs-in on :session_expired, and parses
responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
we actually consume (identity, name, grid, address, lat/lon).
The full XML payload stays in the raw :data jsonb column for
anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
Geocoding API. Only called as a fallback when QRZ has no explicit
<lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
callsign_locations cache, on miss calls Qrz then either uses QRZ's
coords directly or geocodes the formatted address, snaps to an
8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
upserts the result.
Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.
Wiring:
- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
creates qrz_callsigns and callsign_locations with unique indexes
on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
never short-circuits test-level stubs, and supplies dummy QRZ
credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
and GOOGLE_API_KEY from the environment so prod and dev can
configure both upstream keys out of band.
Tests (ported verbatim from gridmap-web):
- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
behavior, upsert on stale, error passthrough, case-insensitive
input
- test/microwaveprop/geocoder_test.exs — success, zero results,
request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
including the QRZ-lat/lon shortcut and the missing-address error
path
All 1294 tests still pass. Credo strict clean.
Replaces the hand-rolled sort/search/pagination on /contacts with
LiveTable.LiveResource. Sortable columns for station1/2, grid1/2,
band, mode, distance_km, qso_timestamp, and inserted_at; searchable
across both stations, both grids, and mode. Custom renderers preserve
the enrichment summary badge (with per-status tooltip), the invalid
flag column, and the qso/inserted_at formatters.
The reciprocal-grouping block has been dropped per the brainstorming
decision — every QSO now shows as its own row, which avoids fighting
live_table's one-row-per-record model. A "View" action links to the
detail page instead of the previous row-click handler (live_table's
stream dom_ids are random UUIDs so row-click by record id is no
longer practical without forking the library).
Test updates:
- sort test switched to ?sort_params[station1]=asc (live_table URL
format; the old ?sort_by/?sort_order params are gone)
- pair-search test removed since live_table does a single ILIKE
across searchable fields and no longer supports the two-callsign
intersection that Radio.list_contacts/1 used to do; single-term
search is exercised instead
- pagination test asserts the presence of "Page" (live_table's
pagination text is "Page N" without the total count)
Also converts the stray cond-with-single-clause in
UserManagementLive.Index's delete handler to if/else (credo fix).
The pending-edits list on the admin review page now sorts through
live_table. Columns for contact, submitted-by user, changed-fields
summary, and submitted timestamp are rendered via 2-arity custom
renderers so the Ecto-preloaded :contact and :user associations stay
accessible — live_table's data_provider path skips select_columns so
the full struct (with preloads) reaches the renderer.
The review drawer, diff table, approve/reject flow, and note textarea
all stay as-is; after approve/reject the page push_patches back to
its current path so live_table re-runs handle_params and the list
refreshes.
Pending count in the header now reads Radio.pending_edit_count/0
instead of length(@edits) since we no longer hold the full list in
socket state.
Replaces the stream-backed static table for approved beacons with
LiveTable.LiveResource, giving users column sorting and full-text
search across callsign and grid. Custom renderers preserve the
frequency/EIRP formatting, keying label, and on-air badge so the
visual output matches the previous daisyUI cells.
A new Beacons.approved_beacons_query/0 returns the base Ecto query
(`where: approved == true`), assigned onto socket.assigns.data_provider
in mount so live_table's handle_params uses it as the root query and
applies sort/search/paginate on top. The leaflet map above the table
keeps its existing phx-hook+phx-update=ignore setup, and the pending-
approval admin section stays as a stream-backed `<.table>` — live_table
is single-resource-per-liveview and the pending list is a distinct
admin concern anyway.
PubSub updates (beacon created/updated/deleted) now push_patch to the
current path so live_table re-runs handle_params and the table
refreshes in place.
Adds an "Open in Path Calculator" action to the contact detail
header. The link navigates to /path with source/destination prefilled
(preferring Maidenhead grids over station callsigns, since grids
are exact and callsigns go through a QRZ + geocoder round-trip)
and the contact's band selected. PathLive's handle_params already
auto-runs the calculation when source + destination are present,
so the user lands straight on the computed path ready to tweak
TX power, antenna heights, and gains.
TerrainAnalysis.analyse was being called with default 0 m antenna
heights on both ends of a QSO's path, which makes the knife-edge
diffraction math pretend the radios are sitting directly on the
terrain surface. Pass 3.048 m (10 ft) for both ant_ht_a and ant_ht_b,
and shift the forward/reverse elevation angles to match. This is
only a display assumption on /contacts/:id — terrain worker paths
computed elsewhere are unaffected.
Hovering the dotted-underlined "callsign" in the Destination label
on /path now surfaces a tooltip explaining that callsign lookup
queries QRZ for the licensee's mailing address and geocodes it, so
the resulting position may differ from where the station actually
operates. Gives users a reason to enter a grid directly when they
know it's more accurate than a QRZ address.
Wire up gurujada/live_table 0.4.1 as a dependency (alongside its
transitive sutra_ui and optional igniter requirement) with the
:live_table config pointing at our Repo and PubSub. Forced oban_web
override so the path-pinned vendored copy still wins over live_table's
hex constraint.
Convert the admin /users page to use LiveTable.LiveResource with
sortable, searchable columns for callsign, name, and email. Custom
renderers preserve the daisyUI admin badge and the pending/confirmed
date cell. Hidden id field stays in the query so the edit and delete
actions can target a record by id.
Updated the delete test to push the delete event directly with an id
payload; the stream dom_ids under live_table are random UUIDs so
tr#users-<id> selectors no longer work.
Collapse the three admin-only links (Users, Contact edits, Oban) into
a single "Admin" dropdown in the horizontal nav so the top bar isn't
cluttered with admin controls for the one admin account. Non-admins
see nothing extra.
The profile page's involving-contacts query now limits to the 100
most recent rows (ordered by qso_timestamp desc). A small note is
shown under the heading when the cap is hit so users know there's
more data they're not seeing.
Clicking anywhere in a row on the /u/:callsign involving-contacts
table now navigates to that contact's detail page, matching the
click affordance on the main /contacts list.
Add a new card on the profile page listing every contact where the
callsign appears as either station1 or station2, case-insensitive,
newest first. Backed by a new `Radio.list_contacts_involving_callsign/1`
query with unit coverage for station1/station2 matching, case handling,
ordering, and blank input.
