Safety net for rows that land via direct DB writes (manual fixes, bulk
imports) and bypass Radio.create_contact's grid-resolution requirement.
Runs every hour on the :admin queue with a 500-row per-invocation cap.
Fills whichever side resolves — if one grid is invalid, the other still
gets populated; distance_km is only written when both positions end up
set.
Widen dedup match to 4-char grid prefix so an upload with a longer grid
(e.g. FN42aa25) finds the existing FN42 contact. Within a match, classify
as a refinement when the upload strictly extends the existing grid or
fills a missing mode; as a contradiction (still skipped) when grids or
modes genuinely disagree. Refinements flow through a new preview bucket,
update the existing row in place, and re-enqueue enrichment when the
position changed.
The /contacts LiveTable export produces a 12-column CSV with label-style
headers (Station 1 / Grid 1 / QSO (UTC) / ...) in a different column
order plus extras (id, distance_km, hrrr_status, flagged_invalid,
inserted_at). The old positional import rejected it with "expected 6 or
7 columns, got 12".
Replace the positional parser with header-driven mapping:
- Parse the header row, normalize each cell (downcase + trim), and look
it up in an alias map that covers both the raw field names and the
export labels.
- Required columns: station1, station2, grid1, grid2, band, qso_timestamp
(mode stays optional). If any are missing the header, return
{:error, {:missing_required_columns, [...]}}.
- Unknown columns are silently ignored so the export's extras don't break
round-trip.
- Per-row validation stays the same (changeset errors still reported by
row number).
Existing 6- or 7-column positional CSVs that already have a header still
work — the header parser recognizes `station1`, `station2`, etc.
directly.
Prod cdo 2.5.1 (Debian) emits named shortnames (tpag10, 2tag2, saip, code11…)
where cdo 2.6.0 (Mac) emits var11/var33/etc. The parser only recognized
varNN, so every prod NARR fetch failed with "no parseable values in cdo
output" and all 350 in-flight jobs discarded.
Switch cdo args from `-outputtab,name,lev,value` to `-outputtab,code,...`.
cdo then outputs the raw numeric NCEP GRIB1 parameter code regardless of
version, and @cdo_code_to_name maps that integer to the semantic name we
already use downstream (TMP/HGT/SPFH/…).
Verified by reproducing on both a 2010 NARR file (fixture) and a 2007 file
fetched live — both now emit identical numeric code rows.
Prod hit "no parseable values in cdo output" for a 2007 record.
Without the actual cdo output it's guesswork whether it's a different
parameter-table format, an empty result, or something else. Capture
stderr (stderr_to_stdout: true) and include first 500 chars of output
so the error message has enough signal to diagnose.
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.
- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
coverage_end so the cron doesn't keep picking post-2014 candidates
- /status progress bar, status-by-type card, and internal map keys now
read "NARR" (data-progress-key="narr", narr_candidates/narr_done).
era5_profiles table stays as the DB landing spot (historical artifact).
- BackfillEnqueueWorker cron now passes "era5" in types so pre-2014
contacts whose hrrr_status is :unavailable actually get NarrFetchWorker
jobs enqueued every 30 min. The :era5 type key still maps internally
to NarrFetchWorker via era5_jobs_for_contact/1.
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
Swap the body of era5_jobs_for_contact/1 to enqueue NarrFetchWorker
instead of Era5FetchWorker, and snap qso_timestamp via
NarrClient.snap_to_analysis_hour/1 so it lands on a 3-hourly NARR
analysis slot (00/03/06/09/12/15/18/21 UTC) — NarrFetchWorker raises
on anything else.
The :era5 type key, the era5_jobs_for_contact function name, and
the era5_status field are kept as historical artifacts — they
still refer to the era5_profiles table, which is reused 1:1 by
NARR. A schema rename is a follow-up PR.
The CDS Era5* workers (Era5FetchWorker, Era5SubmitWorker,
Era5PollWorker, Era5MonthBatchWorker, Era5BatchClient) are now
unreachable from the live submission flow. They stay in tree until
the deletion follow-up PR.
Concurrency 6 in both dev and prod. NARR fetches are network-bound
(byte-range GETs to NCEI) so high concurrency is fine — same shape
as the iemre queue in dev. Required for NarrFetchWorker (added in
the previous commit) to actually pick up jobs.
Per-contact Oban worker that fetches one NARR profile via
NarrClient.fetch_profile_at/2 and inserts it into era5_profiles.
Replaces Era5FetchWorker for pre-2014 contacts. Unlike the ERA5
router, NARR fetches are per-point so this worker does the fetch
+ insert directly — no downstream batch worker, no routing.
Args shape matches Era5FetchWorker: %{"lat", "lon", "valid_time"}.
Lat/lon get snapped to 0.25° (same dedup granularity as ERA5).
valid_time MUST already be on a 3-hourly NARR analysis slot
(00/03/06/09/12/15/18/21 UTC on the hour) — the worker raises
ArgumentError via NarrClient.url_for/1 otherwise. Caller-side
snapping (in ContactWeatherEnqueueWorker) lands in the next task.
Existence check tightened to ±15 minutes (vs ERA5's ±30) since NARR
is exact 3-hourly. On error, logs a warning and returns {:error,
reason} so Oban retries with backoff (max_attempts: 3).
The :narr Oban queue config and the ContactWeatherEnqueueWorker
swap are deliberately NOT touched in this commit — those are the
next two tasks of the plan.
extract_profile_from_file shells out to cdo:
cdo -outputtab,name,lev,value -remapnn,lon=X_lat=Y <merged.grb>
parses the text output (varNNN ↔ NCEP GRIB1 param number lookup),
maps surface records to era5_profiles fields, builds the pressure
profile from each (TMP, HGT, SPFH) triple, and merges in the derived
fields from SoundingParams.derive/1. Returns the same attrs shape as
Era5BatchClient.build_profile_attrs/5 so the existing
bulk_insert_profiles path can ingest NARR rows unchanged.
fetch_profile_at picks records from a fetched inventory, byte-range
fetches each one into a per-record temp file, runs cdo -merge to build
a composite, calls extract_profile_from_file, and cleans up via
try/after.
Drive-by fix: ThetaE.dewpoint_from_spfh/2 returns Kelvin (its existing
test asserts that explicitly). Era5BatchClient was passing that
Kelvin value straight into "dwpc" (Celsius), which would then crash
SoundingParams.compute_refractivity_profile when it called Buck's
saturation-vapor-pressure equation with t=294 instead of t=21. The
bug never surfaced because era5_profiles is empty in production —
no ERA5 backfill has ever succeeded. Fixed in both era5_batch_client
and the new narr_client by subtracting 273.15 in the wrapper. Both
wrappers now have a comment pointing at the K→C conversion.
Test fixture test/fixtures/narr/narr_dfw_2010-06-15_12z.grb is the
spike-captured composite (1.1 MB — Lambert conformal projection
isn't sellonlatbox-subsettable so we can't shrink it further).
parse_inventory walks the wgrib1-format .inv text and returns a
{var, level} -> {byte_offset, length} index, computing each record's
length from the next record's offset (or file_size for the last one).
fetch_inventory does a GET on the .inv URL plus a HEAD on the .grb URL
to learn its size, then delegates to parse_inventory.
Both stubbable via Req.Test (config/test.exs adds narr_req_options
matching the existing era5_req_options / giro_req_options pattern).
Spike fixture at test/fixtures/narr/narr-a_221_20100615_1200_000.inv
backs the parser test — verifies the ("TMP", "2 m above gnd") and
("WVVFLX", "atmos col") records line up with the real bytes.
First slice of the NARR backfill client: url_for/1,
url_for_inventory/1, and snap_to_analysis_hour/1. Pure functions,
no network. Validates that NARR analyses only exist at 3-hourly
slots (00/03/06/09/12/15/18/21 UTC). The byte-range fetch and
cdo-driven point extraction land in follow-up commits per
docs/plans/2026-04-15-merra2-historical-backfill.md.
Replaces the broken ERA5/CDS path with NCEP NARR fetched anonymously
from NCEI. NARR is GRIB1, anonymous HTTPS, byte-range-fetchable, with
1979-01-01 → 2014-10-02 coverage that lines up perfectly against the
HRRR archive start. Variables match the existing era5_profiles schema
1:1 (DPT is a direct surface dewpoint record, no SPFH derivation
needed at the surface).
Phase 1 spike done by hand: byte-range fetched 5 surface records and
3 pressure-level records for DFW 2010-06-15 12Z. wgrib2 turned out to
NOT read GRIB1 messages (rd_grib2_seq_msg, grib1 message ignored, use
wgrib), so the plan now uses cdo for both GRIB1 decoding and Lambert
conformal nearest-neighbor remap. cdo -merge is the right operator
for combining records of different variables — cdo -cat is for time
concat and silently drops mismatched levels, shell cat produces an
invalid composite. cdo emits param numbers as varNNN; mapping table
is in the plan's Architecture section.
Spike values for DFW 2010-06-15 12Z (all physically sane):
TMP_2m=296.94 K (23.8°C) DPT_2m=294.49 K (21.3°C)
HPBL=703 m PRES_sfc=99383 Pa
PWAT=45.1 mm TMP_850mb=291.52 K
HGT_850mb=1547.8 m SPFH_850mb=10.4 g/kg
Plan file kept at the original 2026-04-15-merra2-historical-backfill
filename for git history continuity even though the implementation
is NARR not MERRA-2 (history block at the top documents the pivot).
The persisted grid_data files contain atoms (:base_m, :top_m,
:thickness_m, :min_freq_ghz, :native_min_gradient, :ducts, etc.) that
live in modules which aren't eagerly loaded during application start.
warm_grid_cache_from_latest_profile runs in Application.start/2 before
those modules get pulled in by their first call sites, so [:safe] was
rejecting our own legitimate files at boot — every weather page hit
crashed with "invalid or unsafe external representation of a term".
The whole point of [:safe] is to refuse atoms from untrusted external
ETF input. ProfilesFile.write!/2 in this same module is the only writer
of these files, so the input is trusted by definition — drop the flag.
The test additions round-trip the production data shape including duct
fields as a regression guard.
Prod has 0 era5_profiles rows after ~500 submit cycles: every CDS job
either gets marked "rejected" (likely cap mismatch) or runs for 13+h
without transitioning to successful, so the handle_both_done branch
has literally never fired and the ERA5Batch: stored log line has
never been emitted once. Tighten Era5PollWorker's @stuck_after_seconds
from 18h to 8h so dead rows recycle within a work shift instead of a
full day, and pause the era5 / era5_submit / era5_poll / era5_batch
queues in runtime.exs so existing rows stay put for inspection while
we figure out why CDS is rejecting everything. Flip paused: true back
once the root cause is identified.
Timeline scrubs on /weather were reading a 21 MB NFS ProfilesFile and
then calling SoundingParams.derive on all 92k CONUS points before
filtering to the viewport. Flip the order: filter the raw grid_data map
by bounds first, so derivation work is bounded by the viewport (~4k
points on a DFW box) instead of the full grid.
Add build_grid_cache_rows/3 with an optional bounds argument; the /2
arity is unchanged so PropagationGridWorker still derives the full grid
on f00.
First HF prediction building block. GIRO publishes MUFD — the F2-layer
maximum usable frequency at the 3000 km reference distance — directly
from station measurements, already calibrated against CCIR M(3000)F2
climatology. Rather than recomputing MUF from scratch (which would
require the ~2 MB CCIR coefficient tables), this module treats the
measured MUFD as the anchor and scales it to the actual path distance
using a thin-layer secant-of-incidence ratio.
That keeps the measurement calibration (~3.3× foF2 at our fixture, vs
~5.1× from a naive thin-layer formula) and only uses the thin-layer
math for the relative distance correction.
* adjust_mufd/2 — scales MUFD(3000) to an arbitrary hop distance.
At D=3000 it's a no-op; shorter paths get lower MUF (near-vertical),
longer paths within the single-hop window get higher MUF.
* fot/1 — standard 0.85 × MUF Frequency of Optimum Traffic.
* hf_score/2 — 0-100 band-vs-MUF score with 5-tier curve:
100 (≤ 0.75×MUF) / 80 (FOT) / 50 (≤ MUF) / 20 (fringe) / 0 (dead).
Limitations are in the moduledoc: single-hop only, nearest-station
bias, no LUF/D-layer absorption, no Kp geomagnetic storm modeling.
Those are separate commits once this has bake time.
Not yet wired into PathLive — follow-up commit will add HF bands to
BandConfig and wire the panel.
Adds the second ionospheric data source layer. Polls NOAA SWPC's free
public JSON services every 5 minutes for three products:
* Planetary K-index (1-min cadence) → geomagnetic_observations
* 10.7 cm solar flux (hourly) → solar_flux_observations
* GOES X-ray flux (1-min, 0.1-0.8nm long-wave band only) →
solar_xray_observations
All three products feed HF / Es scoring inputs that the propagation
engine will start using in a follow-up commit:
* Kp drives HF absorption and Es damping (memory notes Kp is inversely
correlated with tropo/Es stability).
* F10.7 is the SSN proxy for ITU-R P.533 HF MUF prediction.
* GOES X-ray long-wave flux is the short-wave fade (SWF) / D-layer
absorption proxy — a C-class flare starts attenuating HF within
seconds of the X-ray peak.
New context `Microwaveprop.SpaceWeather` exposes `upsert_*/1` and
`latest_*/0` per product; `SpaceWeatherFetchWorker` pulls all three
independently so one endpoint's outage doesn't block the others.
Fixtures captured from live SWPC endpoints on 2026-04-15 for the
parser tests.
Not yet wired into scoring — that's a separate commit.
Closes the GIRO → SporadicE → display loop. When a 144 or 440 MHz path
is calculated, PathLive looks up the nearest polled ionosonde's latest
foEs via Ionosphere.nearest_foes/3, runs SporadicE.es_score for the
actual (band, distance) pair, and renders a panel showing:
* Live foEs / foF2 / station-code / timestamp
* Computed single-hop Es MUF for this exact path length
* 0-100 Es score for the selected band
* Human-readable interpretation ("strong opening", "tropo only",
"out of single-hop window", etc.)
Panel only appears for 144/440 (Es isn't relevant on microwave) and
only when the nearest station has data within the last 2 hours — if
ionosonde data is stale or missing, the panel is omitted entirely
rather than shown with misleading values.
Does not touch the grid scorer on /map — a grid cell has no intrinsic
path length, so Es scoring there is conceptually muddy. PathLive is
the right home for path-specific predictions.
Adds the missing link between the GIRO ionosonde data we just started
ingesting and the VHF/UHF band scoring:
* `Microwaveprop.Propagation.SporadicE` — ITU-R P.534-6 / Davies 1990
thin-layer Es MUF (sec(i) · foEs with h=110 km), plus an `es_score/3`
that maps (foEs, target band, hop distance) into a 0-100 single-hop
propagation likelihood. Calibrated against literature: 50 MHz Es at
2000 km needs foEs ≳ 5.5 MHz (routine summer); 144 MHz needs
foEs ≳ 15.8 MHz (rare Jun/Jul peaks only); 440 MHz Es does not occur
at physical foEs values. Multi-hop Es and Es-scatter are separate
factors and explicitly out of scope.
* `Ionosphere.nearest_foes/3` — given a lat/lon, returns the latest
observation from the nearest polled GIRO station (Millstone Hill or
Alpena for now), with a configurable staleness cutoff (default 2h).
Returns `{:error, :stale}` or `{:error, :no_data}` so callers can
choose whether to apply the Es factor at all.
Neither is wired into the grid scorer yet — that's a separate commit
so the integration can be reviewed on its own.
First step toward physics-based HF / sporadic-E scoring. Polls GIRO's
DIDBase tabular endpoint every 10 minutes for two US ionosondes
(Millstone Hill MHJ45, Alpena AL945), parses the plain-text response
into observation rows, and upserts into a new ionosonde_observations
table keyed on (station_code, valid_time).
This gets foEs (E-layer sporadic critical frequency) into the database
as a direct measurement — the key input for predicting 144 MHz Es
openings via ITU-R P.534-6. foF2 + MUFD are the F2-layer inputs for
HF MUF predictions.
Not yet wired into scoring. Boulder/Wallops/Austin/Idaho/Point Arguello
are in the GIRO catalog but were silent when probed — add them back
if/when they come online.
Next steps: SWPC JSON (Kp, F10.7, sunspot), GOES X-ray flux, D-RAP text,
and the P.534-6 Es scoring factor that uses foEs at midpoint for the
144/440 band configs.
Both bands share the physics of low microwave: humidity and refractivity
gradient build ducts the same way, and rain + gas absorption are
effectively zero at VHF/UHF. humidity_effect is :beneficial, rain_k /
rain_alpha are (0, 1), and o2_db_km / h2o_coeff are 0.
Range estimates reflect tropo reality: 2m ducts can reach 2500+ km
under exceptional conditions (e.g. documented Hawaii ↔ California
openings), 70cm a bit less. The seasonal base matches the low-microwave
summer-peak curve.
These configs do NOT capture sporadic-E, meteor scatter, or aurora —
those need ionospheric data (GIRO ionosondes, NOAA SWPC) we don't yet
ingest. The scoring on these bands is therefore "tropo-only" and will
underestimate openings driven by ionospheric modes.
The previous fix only updated the initial mount path. buildLegend gets
re-created on every layer switch via the swap at renderLayer, and that
code path still used the old isMobile-dependent bottomright position,
which clobbered the top-right placement as soon as the user clicked a
different layer.
