Commit graph

298 commits

Author SHA1 Message Date
5fd37a17fd
feat(propagation): per-band weight calibration from full-corpus correlation
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).

- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md
2026-04-18 10:19:26 -05:00
00aad3cb98
feat(auth): self-service password reset
Add a "Forgot your password?" flow off the login page. A 24-hour
reset_password token is emailed on request, the landing page lets the
user pick a new password, and all other tokens for the user are
revoked on success.

The request endpoint returns the same flash regardless of whether the
email matches a user so that attackers can't enumerate accounts.
2026-04-18 08:42:35 -05:00
b8b78f7e57
feat(design): apply NTMS design system tokens
Adopt the handoff design system: navy-slate surfaces with indigo-violet
accent (#6366F1 dark / #5458E3 light), Inter as the UI font, hairline
1px borders, softer corner radii. Add propagation-scale and amateur-band
CSS vars so JS hooks can pull theme-aware colors via getComputedStyle.

Polish: drop table zebra striping for hover-on-base-200 rows with
uppercase small-caps headers, bump .header to text-2xl, tighten nav
spacing with a hairline divider, fix broken text-brand -> text-primary
on the auth screens.

Wire --band-* into contacts_map_hook: resolve once at mount, re-resolve
on data-theme mutation so polylines recolor when the theme toggles.
2026-04-17 17:34:15 -05:00
33f5d4edbe
feat(rainscatter): classify QSO propagation mechanism from common-volume radar
Adds a per-contact enrichment pipeline that determines whether a QSO was
most likely carried by rain scatter, tropospheric ducting, or ordinary
troposcatter — using IEM n0q composite reflectivity sampled inside the
lens-shaped intersection of 400 km-radius disks around each endpoint.

Pieces:
  * Microwaveprop.Propagation.CommonVolume — lens geometry (haversine,
    in-CV test, bbox, area).
  * contact_common_volume_radar table (1:1 per contact) storing
    aggregate dBZ stats inside the CV + radar_status column on contacts.
  * Microwaveprop.Workers.CommonVolumeRadarWorker — Oban :radar queue,
    fetches the n0q frame at QSO time, iterates pixels inside the CV
    bbox, aggregates rain/heavy/core-pixel counts, max/mean dBZ, and
    coverage percentage.
  * Microwaveprop.Propagation.RainScatterClassifier — rule-based mapper
    from (band, distance, radar stats, duct flags) to one of
    :likely_rainscatter | :rainscatter_possible | :tropo_duct |
    :troposcatter | :unknown.
  * ContactWeatherEnqueueWorker learns a :radar enrichment type and
    enqueues the CV worker on contact submission; pre-2014 contacts
    (outside IEM n0q coverage) are pinned to :unavailable.
  * `mix radar_backfill` bulk-enqueues historical contacts with
    --year / --limit / --dry-run.
  * Contact detail page renders a mechanism badge with supporting
    stats (common-volume area, max dBZ, heavy-rain pixel count,
    coverage %).
2026-04-17 15:57:59 -05:00
c09446a517
fix(weather-map): default timeline cursor to the hour closest to now
Previously the map loaded the latest cached valid_time from GridCache,
which can be any +Nh forecast hour the worker has written while walking
f00-f18. Users expect to see current conditions on load, not a forecast
that's ~11 hours out.
2026-04-17 15:57:59 -05:00
b6caffea3e
feat(nav): show last-deployed timestamp
Docker final stage bakes BUILD_TIMESTAMP into /app/BUILD_TIMESTAMP via
build-arg (only consumed after mix compile, so earlier layer caching is
preserved). Application.build_timestamp/0 reads the file, falls back to
DEPLOY_TIMESTAMP env, then to compile time. Navbar renders a compact
"Deployed Xm ago" with the full UTC time in the tooltip.
2026-04-17 13:08:51 -05:00
48cbf40b9d
feat(contacts): show total contact count in list header 2026-04-17 13:08:51 -05:00
fc9d0dc977
feat(agent-discovery): Link/markdown/content-signal/api-catalog/skills
Implements the subset of the isitagentready.com checklist that maps to
real capabilities of this site:

- RFC 8288 Link headers on every browser response advertising
  service-doc (/algo), about, privacy-policy, and sitemap.
- Markdown-for-Agents content negotiation: requests with
  Accept: text/markdown for / or /algo return real markdown
  (curated summary + verbatim algo.md) with x-markdown-tokens hint.
  Other paths still 406 — no synthesizing markdown from LiveView HTML.
- Content-Signal directive in robots.txt declaring
  search=yes, ai-train=no, ai-input=no.
- RFC 9727 API catalog at /.well-known/api-catalog with the sole
  public endpoint (/api/contacts/map) linking status -> /health and
  service-desc -> /openapi.json (a new minimal OpenAPI 3.0 spec).
- Agent Skills Discovery index at /.well-known/agent-skills/index.json
  listing two skill documents (fetch-contacts, read-algorithm) with
  sha256 digests computed at compile time; documents served at
  /.well-known/agent-skills/{name}/SKILL.md.

Skipped (don't match actual site capabilities): Web Bot Auth JWKS,
OAuth/OIDC discovery, MCP server card.
2026-04-17 12:09:05 -05:00
6972feb2c2
perf(contact-map): client-side band filtering via per-band layer groups
Band toggles previously roundtripped through LiveView and rebuilt every
polyline + popup + hover handler on each click. Now the hook builds one
L.LayerGroup per band once after the initial fetch; toggling is just
addLayer/removeLayer on the map (O(bands), not O(contacts)). Checkbox
and All/None events are handled directly in the hook via delegated
document listeners — no LV roundtrip. Callsign filter still roundtrips
(debounced) since it's cross-cutting, and now iterates pre-built
polylines without recreating them.
2026-04-17 12:08:46 -05:00
2a57074a47
feat(contact-detail): use 10 GHz threshold for MHz/GHz band display
Anything below 10 GHz now renders in MHz so 2304/3456/5760 etc. match
amateur microwave convention. 10 GHz and above still render in GHz.
2026-04-17 12:08:38 -05:00
1a8ca44de3
feat(contact-detail): render band in MHz below 1 GHz, GHz above
The detail page was hard-coding GHz formatting, so 50/144/222/432/902
rendered as 0.1/0.4/0.9 GHz instead of the MHz labels every amateur
operator actually uses. Switch to MHz when band < 1000, keep GHz for
microwave bands.

Also surface band and mode in the header subtitle so the key QSO
metadata is visible before SRTM/terrain enrichment completes (the
elevation-profile block that previously rendered them is gated on
that enrichment).
2026-04-17 10:51:54 -05:00
b4a36890da
feat(import): fire confetti when the complete banner appears
Vendors canvas-confetti 1.9.3 and wires an ImportConfetti hook on the
alert-success banner that /imports/:id already renders when
run.status == "complete". The hook uses the "realistic look" preset
from confetti.js.org and runs on mount, so it fires exactly once —
when the banner first appears.
2026-04-17 10:01:18 -05:00
247d066ec5
feat(import): live /imports/:id progress page + gate upload bars on submit
SubmitLive now hands off to the async pipeline:
- confirm_csv/confirm_adif → CsvImport.enqueue/2 → push_navigate to
  /imports/:id. ADIF preview shape matches CSV, so one enqueue call
  serves both.
- allow_upload(auto_upload: false) so bytes only stream when the user
  clicks submit. Progress bar + percentage now gated on
  entry.progress > 0 and < 100 — invisible during file selection,
  only rendered while the actual upload is in-flight.
- Old in-page csv_results / @csv_result / reset_csv removed; the
  /imports/:id page replaces them.

New ImportLive at /imports/🆔 subscribes to 'csv_import:<id>' PubSub,
shows total/processed/imported/refined/error counters in a daisyUI
stats grid, progress bar, status badge, and a collapsed errors table
when error_count > 0. Missing or malformed run_id redirects to /submit
with a flash.

7 new frontend tests (5 ImportLive, 2 updated SubmitLive), 1902 total,
credo clean.
2026-04-17 09:31:22 -05:00
48833f15be
feat(import): async CSV import with progress-tracking schema
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.

- New import_runs table: stores preview rows + running counters (total,
  processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
  Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
  its slice via CsvImport.commit_rows/1, atomically increments counters
  in one update_all, flips status, and broadcasts progress on
  'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
  New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.

Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.

LiveView UI for /imports/:id is a separate commit.
2026-04-17 09:31:22 -05:00
8a969e315c
refactor: normalize pos1/pos2 JSONB key to 'lon' everywhere
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).

- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
  'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
  scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
  (was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
  with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
2026-04-17 09:10:32 -05:00
6f452e7139
feat(map): show CONUS-only banner for visitors outside continental US
Visitor geolocation already flows via the Cloudflare CF-IPLatitude /
CF-IPLongitude headers into the session; MapLive centers the map on
the visitor's location (or DFW as fallback). Add a banner that appears
above the map when those coords are outside the CONUS bbox
(lat 24.5-49.5, lon -125 to -66.5), letting overseas visitors know
propagation data won't be available at their location yet.

Map still centers on the visitor's actual location — this is a note,
not a re-center. Banner is fully absent from the DOM when not needed
(:if conditional).
2026-04-17 08:39:51 -05:00
fbbfe037bc
fix(status): align NARR done-count tolerance with fetcher snap
count_narr_done/0 checked narr_profiles.valid_time within qso_timestamp
± 30 min — but NARR analyses live at 3-hour marks (00Z/03Z/.../21Z) and
NarrClient.snap_to_analysis_hour/1 floors qso_timestamp to the previous
mark before fetching. A QSO at 18:50Z snaps to 18:00Z on the fetcher
side; the profile stored at 18:00Z fell outside the widget's [18:20,
19:20] window, so the status page showed 6/208 done when actual
coverage was 208/208.

Fix: use equality against the Postgres equivalent of the snap
(date_trunc('hour', ts) - make_interval(hours => hour % 3)).
Comment cites NarrClient.snap_to_analysis_hour/1 so future changes keep
them in sync.
2026-04-16 15:14:51 -05:00
ef0c73cb4d
Fix NARR accounting on status page and stop pre-2014 hrrr_status churn
Status page NARR progress counted `hrrr_status = :unavailable` as the
candidate set, which swept in ~13.6K post-2014 HRRR retry rows and
showed "0 / 13,652". Candidate eligibility is qso_timestamp-based, not
hrrr_status-based — switch both the numerator and denominator to
`qso_timestamp < NarrClient.coverage_end()` so pre-2014 rows are
counted whether their hrrr_status is :queued, :unavailable, or
:complete.

ContactWeatherEnqueueWorker: short-circuit hrrr_points_for_contact/1
for pre-coverage timestamps and mark the contact :unavailable when no
HRRR jobs are built. Was producing a ping-pong where reconcile flipped
pre-2014 :queued → :unavailable and the same cron run flipped it back
to :queued from a rebuilt HRRR job that could never land data.
2026-04-16 14:09:45 -05:00
f93f5c9a57
Rename 9cm band 3456 → 3400
The US 9cm amateur allocation was cut from 3300-3500 to 3300-3450 MHz
in 2020, so 3456 (the legacy weak-signal calling frequency) is no
longer in-band. Move the canonical key to 3400, migrate existing
contacts, and keep 3456 input resolving to 3400 via the
nearest-band snap so historical ADIF/CSV logs still import cleanly.
2026-04-16 12:40:38 -05:00
36adf875b7
Add 50/144/222 MHz bands, rename 440→432
Expands submittable-contact bands to include 6m, 2m, 1.25m, and 70cm
(as 432 rather than the old 440 placeholder). Each new band gets an
explicit allocation window in BandResolver.nearest_band so ADIF FREQ
fields near the amateur allocations resolve correctly while 60-900 MHz
frequencies outside those windows are still rejected. Microwave
(>= 900 MHz) snapping is unchanged — nearest-band match across the
full @allowed_bands list.

Also adds BandConfig entries for 50 and 222 (tropo-only config, same
pattern as 144/432). Sporadic-E / F2 / meteor scatter modeling is not
yet in scope — ionosphere data is only used to compute an Es readout
on /path for 50/144/222/432.
2026-04-16 12:36:10 -05:00
11a50bf2c2
Disable PDF export on live_table pages
PDF generation was failing in production. CSV export still works.
2026-04-16 12:25:25 -05:00
bbeb95ff5d
CSV/ADIF import: upsert existing contacts on grid/mode refinement
Widen dedup match to 4-char grid prefix so an upload with a longer grid
(e.g. FN42aa25) finds the existing FN42 contact. Within a match, classify
as a refinement when the upload strictly extends the existing grid or
fills a missing mode; as a contradiction (still skipped) when grids or
modes genuinely disagree. Refinements flow through a new preview bucket,
update the existing row in place, and re-enqueue enrichment when the
position changed.
2026-04-16 10:57:37 -05:00
55d4f289c2
NarrClient.in_coverage?/1 + callsite guards (1979 → 2014-10-02)
Post-2014 contacts with hrrr_status=:unavailable were being dispatched to
NARR, which 404s because NCEI's archive ends 2014-10-02. 14K of the 14.3K
candidates in prod are post-2014 — these are HRRR's responsibility.

- NarrClient.in_coverage?/1 returns true only inside 1979-01-01 → 2014-10-02
- narr_jobs_for_contact and maybe_enqueue_narr (contact_live) bail out of
  coverage returning []
- BackfillEnqueueWorker.type_filter scopes :narr to qso_timestamp <
  coverage_end so the cron doesn't keep picking post-2014 candidates
2026-04-16 09:31:19 -05:00
3604896726
Rename ERA5 → NARR across the codebase
- Schema: Era5Profile → NarrProfile; table renamed via migration
  rename_era5_profiles_to_narr_profiles (ALTER TABLE + rename indexes,
  atomic and instant, no row rewrite)
- Weather helpers: find_nearest_era5/era5_for_contact/era5_profiles_for_path
  → find_nearest_narr/narr_for_contact/narr_profiles_for_path
- BackfillEnqueueWorker: :era5 → :narr type (virtual, no narr_status column);
  reconcile_stale_queued_to_unavailable and status_priority_order now skip
  virtual types via @virtual_types. Fixes the prod crash where the cron tried
  to update a non-existent era5_status column.
- ContactWeatherEnqueueWorker: build_era5_jobs → build_narr_jobs,
  era5_jobs_for_contact → narr_jobs_for_contact
- ContactLive: @era5 → @narr, maybe_enqueue_era5 → maybe_enqueue_narr;
  UI label "ERA5 (0.25°)" → "NARR (32 km)"
- Cron (runtime.exs): args types list now "narr" instead of "era5"
- /status: progress row, status-by-type card, totals.narr_profiles, and
  table-count lookup all target narr_profiles
- Drop obsolete Era5CdsJob schema + era5_cds_jobs table (inspection
  artifact from the retired CDS pipeline; 34 orphan rows in prod)
- Misc docstring/comment cleanups (skew_t, about, wgrib2, propagation_train)

Includes a regression test for the virtual-type crash.
2026-04-16 09:22:23 -05:00
7857d3bc5a
Status page: relabel ERA5 → NARR, include NARR in backfill cron
- /status progress bar, status-by-type card, and internal map keys now
  read "NARR" (data-progress-key="narr", narr_candidates/narr_done).
  era5_profiles table stays as the DB landing spot (historical artifact).
- BackfillEnqueueWorker cron now passes "era5" in types so pre-2014
  contacts whose hrrr_status is :unavailable actually get NarrFetchWorker
  jobs enqueued every 30 min. The :era5 type key still maps internally
  to NarrFetchWorker via era5_jobs_for_contact/1.
2026-04-16 08:57:01 -05:00
1f2a729d40
Drop dead Era5*/CDS code paths
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.

Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs

Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
  table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
  (the symbol still means "historical backfill", just routed to NARR now)

Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).

Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
2026-04-16 08:26:29 -05:00
83ea123e22
PathLive: ionosphere readout panel for VHF paths
Closes the GIRO → SporadicE → display loop. When a 144 or 440 MHz path
is calculated, PathLive looks up the nearest polled ionosonde's latest
foEs via Ionosphere.nearest_foes/3, runs SporadicE.es_score for the
actual (band, distance) pair, and renders a panel showing:

* Live foEs / foF2 / station-code / timestamp
* Computed single-hop Es MUF for this exact path length
* 0-100 Es score for the selected band
* Human-readable interpretation ("strong opening", "tropo only",
  "out of single-hop window", etc.)

Panel only appears for 144/440 (Es isn't relevant on microwave) and
only when the nearest station has data within the last 2 hours — if
ionosonde data is stale or missing, the panel is omitted entirely
rather than shown with misleading values.

Does not touch the grid scorer on /map — a grid cell has no intrinsic
path length, so Es scoring there is conceptually muddy. PathLive is
the right home for path-specific predictions.
2026-04-15 14:48:47 -05:00
33347f7fc2
WeatherMap: legend in top-right, description back in sidebar, play/stop, fix reload loop
Corrections to the previous weather-map pass:

* The "helper" that was supposed to live in the top-right was the
  color legend, not the layer description. Move the Leaflet legend
  control to `topright` (was `bottomright` on desktop).

* Put the layer description back in the desktop sidebar where it
  used to live, now with a "Group · Label" header above the body
  text so it matches the new style in the mockup.

* Add play/stop controls to the weather forecast timeline, ported
  from the propagation map: start plays "now" → end at 1s/step and
  loops; stop returns to the time closest to wall clock. A manual
  click interrupts playback the same way it does on /map.

* Fix the reconnect/reload loop I introduced by making mount do a
  synchronous ProfilesFile read + derive for the full 92k-point
  grid. That was pushing mount past the LV socket join timeout on
  a cold pod, which makes the LV client fall back to a full HTTP
  reload every ~20 seconds (no server-side error). Mount now uses
  the original `latest_weather_grid/1` cache-only hot path and only
  `weather_grid_at/2` (sync disk read) on user timeline scrubs.

* Clean up the playback timer in the hook's `destroyed` callback.
2026-04-15 14:11:53 -05:00
725a7e6bda
WeatherMap: forecast-hour timeline + top-right layer description
Two related UI changes driven by the same data:

* Move the selected-layer description out of the sidebar and into a
  dedicated top-right overlay on the map. The sidebar kept the same
  layer buttons but dropped the descriptive text; the new overlay
  shows group + layer name + description in one card. Mobile still
  shows the description in the collapsible panel since it has no
  top-right free real estate.

* Add a forecast-hour timeline bar at the bottom of the map, matching
  the propagation map. WeatherMapLive enumerates ProfilesFile on mount
  (the grid worker has been persisting f00..f18 on disk since
  commit 07ffcf5), pushes data-valid-times for the JS hook to render
  as buttons, and handles a new select_time event by reading the
  per-hour ProfilesFile on demand via Weather.weather_grid_at/2.

weather_grid_at/2 deliberately skips the GridCache write path for
non-latest hours — caching 18 × 92k rows would add ~300 MB per pod.
A ~2 MB ETF decode per scrub is fast enough for a click.

Subscribes to propagation:pipeline so the timeline picks up new
forecast hours live as they land, without waiting for the full chain
to finish.
2026-04-15 13:57:23 -05:00
790eef1726
PathLive: live-refresh the forecast when new grid scores land
Subscribe to "propagation:updated" on connected mount and, when the
grid worker finishes a forecast-hour step, re-read point_forecast for
the path midpoint using the stashed band_mhz and source/destination.
Terrain, HRRR profiles, and the loss/power budget are independent of
grid scores and stay as-is. Before this, a /path page rendered once at
mount and only refreshed on manual reload.
2026-04-15 13:42:44 -05:00
c3616b31c7
Surface HRRR weather details on /path
Adds an Atmospheric Profile section to the link calculator that shows
the HRRR data actually feeding the prediction: valid/run time, a
per-point table (src/mid/dst) with surface temp, dewpoint, pressure,
HPBL, PWAT, surface refractivity and dN/dh, plus a skew-T log-P chart
for the midpoint vertical profile.

Pulls the skew-T chart function out of contact_live into a shared
component so both LiveViews can render it without duplication.
2026-04-15 11:57:51 -05:00
7fcb2715a0
Enlarge /path forecast chart so detail is legible
Viewbox goes from 300x96 to 1200x240 with aspect-[5/1] so the SVG
fills the card instead of shrinking to a ~300 px strip. Fonts,
gridlines (0/25/50/75/100), and point markers scale up to match,
and the x-axis now shows up to 7 evenly spaced hour labels instead
of just start/middle/end.
2026-04-15 11:09:05 -05:00
222d0fdca3
Fix mojibake in /path loss budget gaseous label
HEEx attribute value syntax (label="...") treats the content as a
raw attribute string, not as an Elixir string literal — so escape
sequences like \u2082 are NOT interpreted and show up verbatim in
the rendered page. Replace with the literal subscript-2 characters
so the label reads 'Gaseous (O₂+H₂O)'.
2026-04-15 10:37:58 -05:00
2d9e5a2d1f
Make the map timeline read .ntms files authoritatively
Two related fixes so the main map reliably picks up new binary
propagation score files as soon as PropagationGridWorker writes them.

1. Propagation.available_valid_times/1 previously preferred ScoreCache
   over ScoresFile, using the cache as an index of what was available.
   The cache is a lazy ETS of whatever hours have been fetched or
   broadcast, which is a strict subset of what's on disk. A new
   forecast hour landing on disk while the cache was warm with older
   entries was invisible to the timeline until the cache happened to
   catch up. Read directly from ScoresFile so the disk store is the
   source of truth.

2. Add Propagation.scores_at_fresh/3 that always reads the .ntms file
   and overwrites the cache entry, and use it from MapLive's
   propagation_updated handler. PropagationGridWorker publishes the
   cache_refresh on `propagation:cache` and the timeline ping on
   `propagation:updated` as separate PubSub broadcasts, so by the time
   MapLive runs through scores_at the ScoreCache GenServer may not
   have processed the refresh yet — fetch_bounds then returns the
   previous chain's bytes. scores_at_fresh takes disk as the source
   of truth for the refresh path and warms the cache as a side effect
   so subsequent readers see the new data.
2026-04-14 17:42:30 -05:00
30c1018400
Extend skew-T plot to cover the full troposphere
Historical contacts showed a skew-T log-P diagram that stopped at 700 mb
because HrrrClient and Era5Client only fetched pressure-level data down
to the top of the boundary layer. The chart canvas already ran up to
100 mb, so the trace clipped mid-atmosphere.

Two complementary fixes:

1. Extend @pressure_levels in HrrrClient and Era5Client with
   650/600/550/500/450/400/350/300/250/200/150/100 mb so new fetches
   cover the full troposphere + lower stratosphere.

2. Prefer the native hybrid-sigma profile for the contact-detail
   skew-T when one has been backfilled for the contact's hour. The
   native profile already stores all 50 hybrid levels up to ~19 km,
   so historical contacts covered by the native backfill get a full
   trace without re-hitting S3. A new HrrrNativeProfile.to_skew_t_profile/1
   converts the parallel arrays into the %{"pres","tmpc","dwpc","hght"}
   list shape the renderer expects, deriving dewpoint from SPFH via the
   Magnus inverse. Weather.find_nearest_native_profile/3 mirrors
   find_nearest_hrrr/3 for the lookup.
2026-04-14 17:25:58 -05:00
92ef6c67fc
Defer beacon coverage estimate until the toggle is flipped on
Previously RangeEstimate.estimate/1 ran on every beacon page mount,
even though the cells data was only rendered after the user flipped
the coverage toggle — a wasted bbox grid pass per page load. Start
with estimate: nil, compute lazily in handle_event("toggle_coverage")
when flipping on, and push the cells payload to the Leaflet hook via
a new load_coverage event. The JS hook no longer reads data-cells on
mount and only builds the cellLayer when the server sends it. Cached
in the socket so re-toggling is instant.
2026-04-14 17:13:05 -05:00
d9ec8a91c3
Stop HRRR badge from wrapping mid-phrase on the contact page
The atmospheric profile header was a single-line flex row with no
wrap, so on narrower viewports the HRRR (3 km) badge and the unit
suffixes were breaking mid-phrase — the badge showed "HRRR (3 /
km)" with borders crossing through the text. Add flex-wrap plus
whitespace-nowrap on each cell so the whole row wraps cleanly
onto a second line instead of shrinking each cell vertically.
2026-04-14 16:59:32 -05:00
7f1a7fb369
Add skew-T log-P diagram to the contact detail page
Render the HRRR pressure-level profile as a full skew-T log-P
diagram on the contact detail page: skewed isotherms, isobars,
dry adiabats, saturation mixing ratio lines, plus the T and Td
traces. Math lives in MicrowavepropWeb.SkewT (Magnus formula,
dry adiabat potential temperature, log-P projection) and is
exercised by a dedicated test module. The atmospheric profile
section now expands by default so the chart is visible without
an extra click.
2026-04-14 16:52:23 -05:00
e3ddf107d9
Gate beacon coverage toggle to 5.76 GHz and above
The estimated current coverage map is only meaningful on bands
where atmospheric attenuation shapes the reach — below 5.76 GHz
it's dominated by free-space loss and misleads. Disable the
toggle, show a helper note explaining the limit, and guard the
handle_event against clients that try to force it.
2026-04-14 16:43:07 -05:00
c7a1e352a7
Show all executing pipeline workers in the status chip
The :propagation Oban queue runs with concurrency 2, so the grid
worker chain step and the 10-minute ASOS nudge can (and do) execute
in parallel. PipelineStatus.current/0 used to return only the
most-recently-started worker, which made the chip label flip to
"ASOS nudge" at every :00 / :10 tick even though HRRR was still
chugging through forecast hours behind it.

Replace the single :detail field with a list of running_detail
maps, each tagged with :grid or :asos so the chip can:

  - Render one sub-line per currently-executing worker
  - Sort them deterministically (grid before asos) so the eye
    doesn't see rows swap
  - Still override the grid entry with the forecast-hour progress
    label when a propagation_pipeline_progress broadcast arrives
2026-04-14 14:04:44 -05:00
0ed47db8b6
Process one forecast hour per PropagationGridWorker perform
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).

The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.

Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.

Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.

Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
2026-04-14 13:24:59 -05:00
fac604b289
Fix ERA5 cap math + stuck threshold, split chip label onto two lines
Era5SubmitWorker was counting era5_cds_jobs rows 1:1 against the CDS
150-job ceiling, but each row holds TWO CDS job IDs (single-level +
pressure-level). Prod hit 74 rows = 148 in-flight jobs and got stuck
in a reject→resubmit spiral. The cap guard now multiplies row count
by 2 to match what CDS sees.

Era5PollWorker @stuck_after_seconds goes from 4h to 18h. CDS
single-level routinely sits at 'accepted' for 12+h under load while
pressure finishes in ~90 min; the 4h threshold was firing on normal
slow runs and feeding the same spiral.

PipelineStatus now returns `label` + `detail` separately so the map
chip can render "Updating propagation" on one line and the
sub-detail ("HRRR run" / "ASOS nudge" / "+3h" / "now") on a second
line below it. running_label/1 is renamed to running_detail/1 and
returns just the sub-part.
2026-04-14 12:06:39 -05:00
434fca2ad1
Show forecast-hour progress on the map pipeline chip
PropagationGridWorker now broadcasts a {:propagation_pipeline_progress,
%{forecast_hour:, valid_time:}} message on the propagation:pipeline
PubSub topic at the start of each process_forecast_hour/4 call.
MapLive subscribes on mount, stashes the last message in the new
:pipeline_progress assign, and clears it on the refresh tick whenever
PipelineStatus flips out of :running.

The chip component consults a new PipelineStatus.running_label/1
helper that formats the payload as "Updating propagation · now" for
f00 and "Updating propagation · +Nh" for f01-f18, falling back to
the base running label until the first progress message arrives.
Covered by unit tests on the formatter and an integration test in
MapLiveTest that seeds an executing grid job, broadcasts progress,
and asserts the rendered chip.
2026-04-14 11:12:51 -05:00
8672ddbafc
Relocate pipeline status chip to the sidebar footer
The running label "Updating propagation (HRRR run)" didn't fit in
the 224 px desktop sidebar alongside the NTMS title block, so it
was being truncated. Move the chip to the bottom of the sidebar
(below the auth menu) where it can use the full sidebar width,
let the text wrap naturally instead of truncating, and drop the
trailing ellipsis. Mobile chip stays at the top since the mobile
layout collapses menu + auth into a dropdown.
2026-04-14 10:41:11 -05:00
554bc9f339
Show aggregated propagation pipeline status on the map
Adds a small chip below the NTMS title in both the mobile and
desktop sidebars of /map that collapses PropagationGridWorker +
AsosAdjustmentWorker Oban state into one of four states —
running / idle / stale / unknown — with a plain-language label
("Updating propagation (HRRR run)…", "Up to date · 12m ago").
The chip refreshes on a 15s timer and whenever the pipeline
broadcasts propagation:updated, so a long HRRR sweep is visible
from the moment a visitor opens the page.
2026-04-14 10:01:34 -05:00
2b323189d0
Add grid squares and weather radar toggles to the weather map
Ports the same two overlay toggles from /map to /weather:
- Maidenhead grid squares via the shared updateGridOverlay helper,
  redrawn on moveend when visible.
- ECCC GeoMet WMS radar (CONUS + Canada dBZ), lazy-created on first
  toggle with a 6-minute tile refresh timer.

Both toggles live in the desktop sidebar below the layer description
and in the mobile pulldown panel. Hook cleanup clears the radar
refresh timer on destroyed().
2026-04-13 16:31:54 -05:00
6c99798ddc
Shrink status page DB stat values so 8-digit counts fit
HRRR Profiles (76M+ rows) was overflowing the stat box at text-2xl.
Drop all four Database stats to text-xl for uniform sizing.
2026-04-13 15:52:37 -05:00
d4b849f976
Show ERA5 fallback on contact detail, fix status page ERA5 progress
Contact detail page now renders an atmospheric profile section backed by
HRRR when available and ERA5 (reanalysis) otherwise, labeled with the
data source. When both sources are missing and HRRR is known-unavailable
(pre-2014 contacts), the Era5FetchWorker is enqueued so the month-tile
backfill gets triggered from a user visit.

Status page ERA5 progress previously computed "complete" as
(total_contacts - hrrr_unavailable_contacts), which treated every contact
with any HRRR status as ERA5-complete — inflating the bar to 76% while
era5_profiles had 0 rows. Replaced with a per-candidate EXISTS query
against era5_profiles using the same 0.15°/30-min tolerance
Weather.find_nearest_era5/3 uses at lookup time.
2026-04-13 13:44:31 -05:00
e46da74472
Show windowed page list in live_table footer
Previously rendered only the current page button. Now renders 1…(current-2)..(current+1)
using just has_next_page, collapsing with an ellipsis past page 4.
2026-04-13 10:02:19 -05:00
a8d4d70abe
live_table polish: border color, control layout, row clicks, pagination
- Tailwind source(none) wasn't scanning deps, so shadcn tokens
  (border-border, bg-muted, text-foreground, ...) used by live_table
  and sutra_ui weren't generated, leaving `border` to fall back to
  currentcolor — a heavy dark border around every table in light mode.
  Add @source directives for deps/live_table/lib and deps/sutra_ui/lib.
- Drop the hardcoded width:100% on the outer .select wrapper so the
  per-page selector's Tailwind w-24 wins instead of stretching across
  the header and overlapping the search box.
- Make whole rows clickable: global click delegator in app.ts forwards
  tr clicks to the first <a href> in the row (the Actions "View" link),
  skipping inner interactive elements. cursor + hover highlight added.
- Replace the default Prev/Next footer with a daisyUI .join/.btn-based
  pager (MicrowavepropWeb.LiveTableFooter), wired up globally via
  :live_table defaults so it applies to every live_table at once.
2026-04-13 08:38:24 -05:00