SubmitLive now hands off to the async pipeline:
- confirm_csv/confirm_adif → CsvImport.enqueue/2 → push_navigate to
/imports/:id. ADIF preview shape matches CSV, so one enqueue call
serves both.
- allow_upload(auto_upload: false) so bytes only stream when the user
clicks submit. Progress bar + percentage now gated on
entry.progress > 0 and < 100 — invisible during file selection,
only rendered while the actual upload is in-flight.
- Old in-page csv_results / @csv_result / reset_csv removed; the
/imports/:id page replaces them.
New ImportLive at /imports/🆔 subscribes to 'csv_import:<id>' PubSub,
shows total/processed/imported/refined/error counters in a daisyUI
stats grid, progress bar, status badge, and a collapsed errors table
when error_count > 0. Missing or malformed run_id redirects to /submit
with a flash.
7 new frontend tests (5 ImportLive, 2 updated SubmitLive), 1902 total,
credo clean.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
Visitor geolocation already flows via the Cloudflare CF-IPLatitude /
CF-IPLongitude headers into the session; MapLive centers the map on
the visitor's location (or DFW as fallback). Add a banner that appears
above the map when those coords are outside the CONUS bbox
(lat 24.5-49.5, lon -125 to -66.5), letting overseas visitors know
propagation data won't be available at their location yet.
Map still centers on the visitor's actual location — this is a note,
not a re-center. Banner is fully absent from the DOM when not needed
(:if conditional).
parse_cdo_row/1 assumed cdo -outputtab,code,lev,value would always emit
numeric codes in the first column, but Debian's cdo 2.5.1 (the version
in the prod container) emits mixed output:
- 'codeNN' with a literal 'code' prefix for most GRIB1 parameters
- Short GRIB1 names for a few: 2tag2 (PRES sfc), saip (DPT 2m),
tpag10 (HGT)
Result: every row fell through to :error, yielding a 'no parseable
values' error and 986 discarded NarrFetchWorker jobs.
Fix: normalize the code token by stripping 'code' prefix when present
and mapping known 2.5.1 short names. UGRD/VGRD intentionally omitted
(no discard evidence yet) — comment documents the add-when-they-surface
plan.
Also exposed parse_cdo_outputtab/1 as @doc false for testability.
count_narr_done/0 checked narr_profiles.valid_time within qso_timestamp
± 30 min — but NARR analyses live at 3-hour marks (00Z/03Z/.../21Z) and
NarrClient.snap_to_analysis_hour/1 floors qso_timestamp to the previous
mark before fetching. A QSO at 18:50Z snaps to 18:00Z on the fetcher
side; the profile stored at 18:00Z fell outside the widget's [18:20,
19:20] window, so the status page showed 6/208 done when actual
coverage was 208/208.
Fix: use equality against the Postgres equivalent of the snap
(date_trunc('hour', ts) - make_interval(hours => hour % 3)).
Comment cites NarrClient.snap_to_analysis_hour/1 so future changes keep
them in sync.
Backtest branch's unknown-feature guard was dead code: the if block's
{:error, _} return was discarded and execution fell through, running the
backtest with a just-created atom and writing a report for nonsense
features. Two compounding issues:
- function_exported?/3 returns false for modules not yet loaded in the
BEAM, so even valid feature names triggered the warning path on a
cold VM.
- String.to_atom/1 on caller-supplied strings created unbounded atoms.
Fix: Code.ensure_loaded?(Features) before the check, String.to_existing_atom/1
with rescue (Features exports __info__(:functions) so every valid function
atom already exists), and an explicit early return via a with pipeline so
the error tuple actually leaves the worker.
FreshnessMonitor's comment claimed 'Oban unique constraint on
PropagationGridWorker prevents duplicates' — but the worker declared
no unique: option. During a long outage the monitor's 5-min stale-check
tick would pile up identical jobs (a 2h outage = 24 stacked jobs, each
doing the full f00-f18 HRRR chain).
Put the dedup on the worker (vs. the monitor's insert call) so anything
enqueuing it benefits — FreshnessMonitor, the hourly cron, or a manual
mix propagation_grid run.
unique: [period: 3600, states: [:available, :scheduled, :executing,
:retryable]] aligns with the hourly cron cadence. The seed job args
(%{}) collapse with themselves while chain-step jobs keep distinct
run_time/forecast_hour and remain enqueuable.
build_messages_per_message/3 was slicing the wgrib2 -lola bin output as
a packed float32 stream, ignoring the 4-byte record-length header + 4-byte
trailer that Fortran unformatted records prepend/append around each
message's data block. First value of message 0 came back as ~2.24e-44
(little-endian float32 of the int-16 record length), and every subsequent
message was shifted by 8 bytes so values spilled into neighbouring cells.
Fix: propagate the correct offset math already used by parse_lola_binary/3
(record_overhead = 8, stride = bytes_per_message + 8, data_offset =
msg_idx * stride + 4).
Flipped the characterization test from metadata-only to per-cell physical
asserts (200 K < TMP, DPT < 340 K; DPT <= TMP) and a 0.01 K cross-check
against extract_grid/3 output on the same fixture + bbox. Latent bug —
no production callers of extract_grid_messages*.
Same fix as rtma_client: MrmsClient had a hardcoded req_options/0 —
switched to a Keyword.merge with Application.get_env so tests can stub
HTTP. 13 new characterization tests: cache put/get/clear/broadcast and
worker listing/download/up-to-date paths. Full GRIB2 parse happy path
deferred to the wgrib2 coverage task.
RtmaClient was the only weather client without a Req.Test plug seam —
added the standard req_options() helper matching NarrClient/HrrrClient/
etc. so its HTTP calls can be stubbed in test.
17 characterization tests: URL construction, idx fetch 404/503, hour
truncation, malformed idx, range-GET error propagation, worker's
existing-row short-circuit, client-error propagation. GRIB2 byte
parsing paths deferred to the wgrib2 coverage task.
Status page NARR progress counted `hrrr_status = :unavailable` as the
candidate set, which swept in ~13.6K post-2014 HRRR retry rows and
showed "0 / 13,652". Candidate eligibility is qso_timestamp-based, not
hrrr_status-based — switch both the numerator and denominator to
`qso_timestamp < NarrClient.coverage_end()` so pre-2014 rows are
counted whether their hrrr_status is :queued, :unavailable, or
:complete.
ContactWeatherEnqueueWorker: short-circuit hrrr_points_for_contact/1
for pre-coverage timestamps and mark the contact :unavailable when no
HRRR jobs are built. Was producing a ping-pong where reconcile flipped
pre-2014 :queued → :unavailable and the same cron run flipped it back
to :queued from a rebuilt HRRR job that could never land data.
Makes the 'always recompute' intent explicit by removing the dead-code
short-circuit in add_distance_change — the candidate query already
requires at least one nil pos, so any previously stored distance is
tied to a stale grid/pos pair and must be overwritten.
10K+ HRRR-pending contacts were starving the 200-ish NARR candidates
out of the 500 per-run slot each 30 min because NARR candidates sort
last by the :hrrr_status priority. Drop the cap so every eligible
contact gets enqueued on each cron run; queue concurrency still paces
the actual fetches. Worker keeps supporting an explicit "limit" arg
for ad-hoc runs and tests.
The US 9cm amateur allocation was cut from 3300-3500 to 3300-3450 MHz
in 2020, so 3456 (the legacy weak-signal calling frequency) is no
longer in-band. Move the canonical key to 3400, migrate existing
contacts, and keep 3456 input resolving to 3400 via the
nearest-band snap so historical ADIF/CSV logs still import cleanly.
Expands submittable-contact bands to include 6m, 2m, 1.25m, and 70cm
(as 432 rather than the old 440 placeholder). Each new band gets an
explicit allocation window in BandResolver.nearest_band so ADIF FREQ
fields near the amateur allocations resolve correctly while 60-900 MHz
frequencies outside those windows are still rejected. Microwave
(>= 900 MHz) snapping is unchanged — nearest-band match across the
full @allowed_bands list.
Also adds BandConfig entries for 50 and 222 (tropo-only config, same
pattern as 144/432). Sporadic-E / F2 / meteor scatter modeling is not
yet in scope — ionosphere data is only used to compute an Es readout
on /path for 50/144/222/432.
Safety net for rows that land via direct DB writes (manual fixes, bulk
imports) and bypass Radio.create_contact's grid-resolution requirement.
Runs every hour on the :admin queue with a 500-row per-invocation cap.
Fills whichever side resolves — if one grid is invalid, the other still
gets populated; distance_km is only written when both positions end up
set.
Widen dedup match to 4-char grid prefix so an upload with a longer grid
(e.g. FN42aa25) finds the existing FN42 contact. Within a match, classify
as a refinement when the upload strictly extends the existing grid or
fills a missing mode; as a contradiction (still skipped) when grids or
modes genuinely disagree. Refinements flow through a new preview bucket,
update the existing row in place, and re-enqueue enrichment when the
position changed.
The /contacts LiveTable export produces a 12-column CSV with label-style
headers (Station 1 / Grid 1 / QSO (UTC) / ...) in a different column
order plus extras (id, distance_km, hrrr_status, flagged_invalid,
inserted_at). The old positional import rejected it with "expected 6 or
7 columns, got 12".
Replace the positional parser with header-driven mapping:
- Parse the header row, normalize each cell (downcase + trim), and look
it up in an alias map that covers both the raw field names and the
export labels.
- Required columns: station1, station2, grid1, grid2, band, qso_timestamp
(mode stays optional). If any are missing the header, return
{:error, {:missing_required_columns, [...]}}.
- Unknown columns are silently ignored so the export's extras don't break
round-trip.
- Per-row validation stays the same (changeset errors still reported by
row number).
Existing 6- or 7-column positional CSVs that already have a header still
work — the header parser recognizes `station1`, `station2`, etc.
directly.
Prod cdo 2.5.1 (Debian) emits named shortnames (tpag10, 2tag2, saip, code11…)
where cdo 2.6.0 (Mac) emits var11/var33/etc. The parser only recognized
varNN, so every prod NARR fetch failed with "no parseable values in cdo
output" and all 350 in-flight jobs discarded.
Switch cdo args from `-outputtab,name,lev,value` to `-outputtab,code,...`.
cdo then outputs the raw numeric NCEP GRIB1 parameter code regardless of
version, and @cdo_code_to_name maps that integer to the semantic name we
already use downstream (TMP/HGT/SPFH/…).
Verified by reproducing on both a 2010 NARR file (fixture) and a 2007 file
fetched live — both now emit identical numeric code rows.
Prod hit "no parseable values in cdo output" for a 2007 record.
Without the actual cdo output it's guesswork whether it's a different
parameter-table format, an empty result, or something else. Capture
stderr (stderr_to_stdout: true) and include first 500 chars of output
so the error message has enough signal to diagnose.
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.
- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
coverage_end so the cron doesn't keep picking post-2014 candidates
- /status progress bar, status-by-type card, and internal map keys now
read "NARR" (data-progress-key="narr", narr_candidates/narr_done).
era5_profiles table stays as the DB landing spot (historical artifact).
- BackfillEnqueueWorker cron now passes "era5" in types so pre-2014
contacts whose hrrr_status is :unavailable actually get NarrFetchWorker
jobs enqueued every 30 min. The :era5 type key still maps internally
to NarrFetchWorker via era5_jobs_for_contact/1.
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
Swap the body of era5_jobs_for_contact/1 to enqueue NarrFetchWorker
instead of Era5FetchWorker, and snap qso_timestamp via
NarrClient.snap_to_analysis_hour/1 so it lands on a 3-hourly NARR
analysis slot (00/03/06/09/12/15/18/21 UTC) — NarrFetchWorker raises
on anything else.
The :era5 type key, the era5_jobs_for_contact function name, and
the era5_status field are kept as historical artifacts — they
still refer to the era5_profiles table, which is reused 1:1 by
NARR. A schema rename is a follow-up PR.
The CDS Era5* workers (Era5FetchWorker, Era5SubmitWorker,
Era5PollWorker, Era5MonthBatchWorker, Era5BatchClient) are now
unreachable from the live submission flow. They stay in tree until
the deletion follow-up PR.
Per-contact Oban worker that fetches one NARR profile via
NarrClient.fetch_profile_at/2 and inserts it into era5_profiles.
Replaces Era5FetchWorker for pre-2014 contacts. Unlike the ERA5
router, NARR fetches are per-point so this worker does the fetch
+ insert directly — no downstream batch worker, no routing.
Args shape matches Era5FetchWorker: %{"lat", "lon", "valid_time"}.
Lat/lon get snapped to 0.25° (same dedup granularity as ERA5).
valid_time MUST already be on a 3-hourly NARR analysis slot
(00/03/06/09/12/15/18/21 UTC on the hour) — the worker raises
ArgumentError via NarrClient.url_for/1 otherwise. Caller-side
snapping (in ContactWeatherEnqueueWorker) lands in the next task.
Existence check tightened to ±15 minutes (vs ERA5's ±30) since NARR
is exact 3-hourly. On error, logs a warning and returns {:error,
reason} so Oban retries with backoff (max_attempts: 3).
The :narr Oban queue config and the ContactWeatherEnqueueWorker
swap are deliberately NOT touched in this commit — those are the
next two tasks of the plan.
extract_profile_from_file shells out to cdo:
cdo -outputtab,name,lev,value -remapnn,lon=X_lat=Y <merged.grb>
parses the text output (varNNN ↔ NCEP GRIB1 param number lookup),
maps surface records to era5_profiles fields, builds the pressure
profile from each (TMP, HGT, SPFH) triple, and merges in the derived
fields from SoundingParams.derive/1. Returns the same attrs shape as
Era5BatchClient.build_profile_attrs/5 so the existing
bulk_insert_profiles path can ingest NARR rows unchanged.
fetch_profile_at picks records from a fetched inventory, byte-range
fetches each one into a per-record temp file, runs cdo -merge to build
a composite, calls extract_profile_from_file, and cleans up via
try/after.
Drive-by fix: ThetaE.dewpoint_from_spfh/2 returns Kelvin (its existing
test asserts that explicitly). Era5BatchClient was passing that
Kelvin value straight into "dwpc" (Celsius), which would then crash
SoundingParams.compute_refractivity_profile when it called Buck's
saturation-vapor-pressure equation with t=294 instead of t=21. The
bug never surfaced because era5_profiles is empty in production —
no ERA5 backfill has ever succeeded. Fixed in both era5_batch_client
and the new narr_client by subtracting 273.15 in the wrapper. Both
wrappers now have a comment pointing at the K→C conversion.
Test fixture test/fixtures/narr/narr_dfw_2010-06-15_12z.grb is the
spike-captured composite (1.1 MB — Lambert conformal projection
isn't sellonlatbox-subsettable so we can't shrink it further).
parse_inventory walks the wgrib1-format .inv text and returns a
{var, level} -> {byte_offset, length} index, computing each record's
length from the next record's offset (or file_size for the last one).
fetch_inventory does a GET on the .inv URL plus a HEAD on the .grb URL
to learn its size, then delegates to parse_inventory.
Both stubbable via Req.Test (config/test.exs adds narr_req_options
matching the existing era5_req_options / giro_req_options pattern).
Spike fixture at test/fixtures/narr/narr-a_221_20100615_1200_000.inv
backs the parser test — verifies the ("TMP", "2 m above gnd") and
("WVVFLX", "atmos col") records line up with the real bytes.
First slice of the NARR backfill client: url_for/1,
url_for_inventory/1, and snap_to_analysis_hour/1. Pure functions,
no network. Validates that NARR analyses only exist at 3-hourly
slots (00/03/06/09/12/15/18/21 UTC). The byte-range fetch and
cdo-driven point extraction land in follow-up commits per
docs/plans/2026-04-15-merra2-historical-backfill.md.
The persisted grid_data files contain atoms (:base_m, :top_m,
:thickness_m, :min_freq_ghz, :native_min_gradient, :ducts, etc.) that
live in modules which aren't eagerly loaded during application start.
warm_grid_cache_from_latest_profile runs in Application.start/2 before
those modules get pulled in by their first call sites, so [:safe] was
rejecting our own legitimate files at boot — every weather page hit
crashed with "invalid or unsafe external representation of a term".
The whole point of [:safe] is to refuse atoms from untrusted external
ETF input. ProfilesFile.write!/2 in this same module is the only writer
of these files, so the input is trusted by definition — drop the flag.
The test additions round-trip the production data shape including duct
fields as a regression guard.
Prod has 0 era5_profiles rows after ~500 submit cycles: every CDS job
either gets marked "rejected" (likely cap mismatch) or runs for 13+h
without transitioning to successful, so the handle_both_done branch
has literally never fired and the ERA5Batch: stored log line has
never been emitted once. Tighten Era5PollWorker's @stuck_after_seconds
from 18h to 8h so dead rows recycle within a work shift instead of a
full day, and pause the era5 / era5_submit / era5_poll / era5_batch
queues in runtime.exs so existing rows stay put for inspection while
we figure out why CDS is rejecting everything. Flip paused: true back
once the root cause is identified.
Timeline scrubs on /weather were reading a 21 MB NFS ProfilesFile and
then calling SoundingParams.derive on all 92k CONUS points before
filtering to the viewport. Flip the order: filter the raw grid_data map
by bounds first, so derivation work is bounded by the viewport (~4k
points on a DFW box) instead of the full grid.
Add build_grid_cache_rows/3 with an optional bounds argument; the /2
arity is unchanged so PropagationGridWorker still derives the full grid
on f00.
First HF prediction building block. GIRO publishes MUFD — the F2-layer
maximum usable frequency at the 3000 km reference distance — directly
from station measurements, already calibrated against CCIR M(3000)F2
climatology. Rather than recomputing MUF from scratch (which would
require the ~2 MB CCIR coefficient tables), this module treats the
measured MUFD as the anchor and scales it to the actual path distance
using a thin-layer secant-of-incidence ratio.
That keeps the measurement calibration (~3.3× foF2 at our fixture, vs
~5.1× from a naive thin-layer formula) and only uses the thin-layer
math for the relative distance correction.
* adjust_mufd/2 — scales MUFD(3000) to an arbitrary hop distance.
At D=3000 it's a no-op; shorter paths get lower MUF (near-vertical),
longer paths within the single-hop window get higher MUF.
* fot/1 — standard 0.85 × MUF Frequency of Optimum Traffic.
* hf_score/2 — 0-100 band-vs-MUF score with 5-tier curve:
100 (≤ 0.75×MUF) / 80 (FOT) / 50 (≤ MUF) / 20 (fringe) / 0 (dead).
Limitations are in the moduledoc: single-hop only, nearest-station
bias, no LUF/D-layer absorption, no Kp geomagnetic storm modeling.
Those are separate commits once this has bake time.
Not yet wired into PathLive — follow-up commit will add HF bands to
BandConfig and wire the panel.
Adds the second ionospheric data source layer. Polls NOAA SWPC's free
public JSON services every 5 minutes for three products:
* Planetary K-index (1-min cadence) → geomagnetic_observations
* 10.7 cm solar flux (hourly) → solar_flux_observations
* GOES X-ray flux (1-min, 0.1-0.8nm long-wave band only) →
solar_xray_observations
All three products feed HF / Es scoring inputs that the propagation
engine will start using in a follow-up commit:
* Kp drives HF absorption and Es damping (memory notes Kp is inversely
correlated with tropo/Es stability).
* F10.7 is the SSN proxy for ITU-R P.533 HF MUF prediction.
* GOES X-ray long-wave flux is the short-wave fade (SWF) / D-layer
absorption proxy — a C-class flare starts attenuating HF within
seconds of the X-ray peak.
New context `Microwaveprop.SpaceWeather` exposes `upsert_*/1` and
`latest_*/0` per product; `SpaceWeatherFetchWorker` pulls all three
independently so one endpoint's outage doesn't block the others.
Fixtures captured from live SWPC endpoints on 2026-04-15 for the
parser tests.
Not yet wired into scoring — that's a separate commit.
Closes the GIRO → SporadicE → display loop. When a 144 or 440 MHz path
is calculated, PathLive looks up the nearest polled ionosonde's latest
foEs via Ionosphere.nearest_foes/3, runs SporadicE.es_score for the
actual (band, distance) pair, and renders a panel showing:
* Live foEs / foF2 / station-code / timestamp
* Computed single-hop Es MUF for this exact path length
* 0-100 Es score for the selected band
* Human-readable interpretation ("strong opening", "tropo only",
"out of single-hop window", etc.)
Panel only appears for 144/440 (Es isn't relevant on microwave) and
only when the nearest station has data within the last 2 hours — if
ionosonde data is stale or missing, the panel is omitted entirely
rather than shown with misleading values.
Does not touch the grid scorer on /map — a grid cell has no intrinsic
path length, so Es scoring there is conceptually muddy. PathLive is
the right home for path-specific predictions.
Adds the missing link between the GIRO ionosonde data we just started
ingesting and the VHF/UHF band scoring:
* `Microwaveprop.Propagation.SporadicE` — ITU-R P.534-6 / Davies 1990
thin-layer Es MUF (sec(i) · foEs with h=110 km), plus an `es_score/3`
that maps (foEs, target band, hop distance) into a 0-100 single-hop
propagation likelihood. Calibrated against literature: 50 MHz Es at
2000 km needs foEs ≳ 5.5 MHz (routine summer); 144 MHz needs
foEs ≳ 15.8 MHz (rare Jun/Jul peaks only); 440 MHz Es does not occur
at physical foEs values. Multi-hop Es and Es-scatter are separate
factors and explicitly out of scope.
* `Ionosphere.nearest_foes/3` — given a lat/lon, returns the latest
observation from the nearest polled GIRO station (Millstone Hill or
Alpena for now), with a configurable staleness cutoff (default 2h).
Returns `{:error, :stale}` or `{:error, :no_data}` so callers can
choose whether to apply the Es factor at all.
Neither is wired into the grid scorer yet — that's a separate commit
so the integration can be reviewed on its own.
First step toward physics-based HF / sporadic-E scoring. Polls GIRO's
DIDBase tabular endpoint every 10 minutes for two US ionosondes
(Millstone Hill MHJ45, Alpena AL945), parses the plain-text response
into observation rows, and upserts into a new ionosonde_observations
table keyed on (station_code, valid_time).
This gets foEs (E-layer sporadic critical frequency) into the database
as a direct measurement — the key input for predicting 144 MHz Es
openings via ITU-R P.534-6. foF2 + MUFD are the F2-layer inputs for
HF MUF predictions.
Not yet wired into scoring. Boulder/Wallops/Austin/Idaho/Point Arguello
are in the GIRO catalog but were silent when probed — add them back
if/when they come online.
Next steps: SWPC JSON (Kp, F10.7, sunspot), GOES X-ray flux, D-RAP text,
and the P.534-6 Es scoring factor that uses foEs at midpoint for the
144/440 band configs.
Both bands share the physics of low microwave: humidity and refractivity
gradient build ducts the same way, and rain + gas absorption are
effectively zero at VHF/UHF. humidity_effect is :beneficial, rain_k /
rain_alpha are (0, 1), and o2_db_km / h2o_coeff are 0.
Range estimates reflect tropo reality: 2m ducts can reach 2500+ km
under exceptional conditions (e.g. documented Hawaii ↔ California
openings), 70cm a bit less. The seasonal base matches the low-microwave
summer-peak curve.
These configs do NOT capture sporadic-E, meteor scatter, or aurora —
those need ionospheric data (GIRO ionosondes, NOAA SWPC) we don't yet
ingest. The scoring on these bands is therefore "tropo-only" and will
underestimate openings driven by ionospheric modes.
Corrections to the previous weather-map pass:
* The "helper" that was supposed to live in the top-right was the
color legend, not the layer description. Move the Leaflet legend
control to `topright` (was `bottomright` on desktop).
* Put the layer description back in the desktop sidebar where it
used to live, now with a "Group · Label" header above the body
text so it matches the new style in the mockup.
* Add play/stop controls to the weather forecast timeline, ported
from the propagation map: start plays "now" → end at 1s/step and
loops; stop returns to the time closest to wall clock. A manual
click interrupts playback the same way it does on /map.
* Fix the reconnect/reload loop I introduced by making mount do a
synchronous ProfilesFile read + derive for the full 92k-point
grid. That was pushing mount past the LV socket join timeout on
a cold pod, which makes the LV client fall back to a full HTTP
reload every ~20 seconds (no server-side error). Mount now uses
the original `latest_weather_grid/1` cache-only hot path and only
`weather_grid_at/2` (sync disk read) on user timeline scrubs.
* Clean up the playback timer in the hook's `destroyed` callback.
Two related UI changes driven by the same data:
* Move the selected-layer description out of the sidebar and into a
dedicated top-right overlay on the map. The sidebar kept the same
layer buttons but dropped the descriptive text; the new overlay
shows group + layer name + description in one card. Mobile still
shows the description in the collapsible panel since it has no
top-right free real estate.
* Add a forecast-hour timeline bar at the bottom of the map, matching
the propagation map. WeatherMapLive enumerates ProfilesFile on mount
(the grid worker has been persisting f00..f18 on disk since
commit 07ffcf5), pushes data-valid-times for the JS hook to render
as buttons, and handles a new select_time event by reading the
per-hour ProfilesFile on demand via Weather.weather_grid_at/2.
weather_grid_at/2 deliberately skips the GridCache write path for
non-latest hours — caching 18 × 92k rows would add ~300 MB per pod.
A ~2 MB ETF decode per scrub is fast enough for a click.
Subscribes to propagation:pipeline so the timeline picks up new
forecast hours live as they land, without waiting for the full chain
to finish.
Subscribe to "propagation:updated" on connected mount and, when the
grid worker finishes a forecast-hour step, re-read point_forecast for
the path midpoint using the stashed band_mhz and source/destination.
Terrain, HRRR profiles, and the loss/power budget are independent of
grid scores and stay as-is. Before this, a /path page rendered once at
mount and only refreshed on manual reload.
A brief CDS network blip on 2026-04-15 returned HTTP 5xx from check_status
for multiple concurrently-polling workers. The old `{:error, _}` branches
consumed attempts, max_attempts=10 burned out in seconds, and 13 rows
were discarded with no cleanup — orphaned with no active poll worker,
including two rows that had both CDS legs done and were ready to decode.
Result: zero era5_profiles ever inserted despite 59 tile-months in flight.
Transient HTTP errors are now a `{:snooze, 60}` so a CDS outage pauses
polling instead of killing it. Terminal states (:failed, :rejected,
:not_found, :stuck) still drive the fail/resubmit paths.
The grid worker stopped writing HRRR profiles to the DB on April 14
when the forecast grid moved to /data/scores binary files, but the
24/72h-windowed prune left every is_grid_point=true row older than
72 hours sitting in the table (billions of rows across every
partition back to 2016, ~90% of the 146 GB table size). Nothing
reads them anymore.
Replace prune_old_grid_profiles/0 with purge_grid_point_profiles/0,
which walks every hrrr_profiles partition directly and deletes
is_grid_point=true rows regardless of age. Per-partition DELETEs
avoid a parent-table full scan. QSO-linked rows (is_grid_point=false)
are untouched — those still back contact enrichment and /path.
Call the new purge from the grid worker's end-of-chain hook so any
grid-point row that somehow slips back in gets cleaned up within the
next hour. Add `mix hrrr.purge_grid_points` for a one-shot historical
cleanup.
Adds an Atmospheric Profile section to the link calculator that shows
the HRRR data actually feeding the prediction: valid/run time, a
per-point table (src/mid/dst) with surface temp, dewpoint, pressure,
HPBL, PWAT, surface refractivity and dN/dh, plus a skew-T log-P chart
for the midpoint vertical profile.
Pulls the skew-T chart function out of contact_live into a shared
component so both LiveViews can render it without duplication.
Viewbox goes from 300x96 to 1200x240 with aspect-[5/1] so the SVG
fills the card instead of shrinking to a ~300 px strip. Fonts,
gridlines (0/25/50/75/100), and point markers scale up to match,
and the x-axis now shows up to 7 evenly spaced hour labels instead
of just start/middle/end.
point_forecast used a strict \`>= now\` cutoff, so the leftmost
"now" sample was always dropped — the HRRR publishing lag means the
newest analysis file is typically 30–60 min behind the wall clock,
which made the filter evict the very hour the chart most wanted to
anchor on. If the chain ever fell behind by even one step the whole
forecast went empty and the chart disappeared.
Use the same window as available_valid_times: keep everything from
one hour before now onward, and fall back to the single newest file
when nothing is fresh enough. Both the cache-path and store-path
flows go through a shared forecast_window helper so the behavior
stays in lockstep with the map timeline.
HEEx attribute value syntax (label="...") treats the content as a
raw attribute string, not as an Elixir string literal — so escape
sequences like \u2082 are NOT interpreted and show up verbatim in
the rendered page. Replace with the literal subscript-2 characters
so the label reads 'Gaseous (O₂+H₂O)'.
The "Updating propagation +Nh" chip was double-misleading in prod:
the label's +Nh was relative to the HRRR run time (which lags wall
clock by ~2h), and the progress broadcast fired before the hour was
fetched + persisted — so the chip could read "+8h" while the map
timeline only extended to +4h.
Reframe the label as "through now" / "through +Nh" / "through Nh ago"
by computing the offset from valid_time → now in PipelineStatus.
running_detail, so it matches the semantic the user reads off the
timeline. Move the PubSub broadcast in PropagationGridWorker to fire
after Propagation.replace_scores/2 succeeds, so the chip only
advances once that forecast hour is readable from ScoresFile.
