- Extract PathParser.valid_callsign?/1 and reuse from the calibration
Mix task instead of duplicating the regex.
- Document raise behavior on Aprs.recent_packets_with_paths/1 and
Aprs.station_positions/1 (already noted in moduledoc; @doc reinforces
for callers reading the function head).
- Note in the BandConfig guard comment why it sits before the Oban pause
(so a misconfig exits without leaving queues paused).
- Reword the band-mismatched-negatives note as a v1 limitation (no TODO
tag — codebase convention is zero TODOs).
- Print '<computed> / <attempted>' for both positive (HRRR-coverage) and
negative (random-baseline → HRRR) factor compute stages so an operator
can see how much of the corpus survived each filter.
- Guard at task entry: Mix.raise if BandConfig has no 144 MHz entry.
Without it the Recalibrator silently falls back to 10 GHz physics.
- random_baseline now draws from sample_size: max(n, 5_000) so small-n
fits don't cluster geographically.
- Wrap body in try/after Oban.resume_all_queues so iex -S mix workflows
don't leave Oban paused after the task returns.
Pulls verified RF hops from aprs.me's 24h packet retention, computes
factor vectors at each hop's midpoint via the existing Recalibrator,
and runs gradient descent against random-baseline negatives.
Dry-run: prints proposed weights but does NOT mutate band_config.ex.
Refuses to fit on <50 positive or negative samples.
Stages 6, 7, 10 of the HRDPS plan, plus the Elixir read-side merge
that makes Rust's `.hrdps.prop` writes show up on /map.
Read side:
- `ScoresFile.path_for_hrdps/2` + `read_hrdps/2` — companions to the
HRRR equivalents. Same decode path; different filename.
- `ScoresFile.read_bounds/3` now reads HRRR + HRDPS files and
concatenates the cells. Files are disjoint by construction
(Grid.hrdps_only_points excludes CONUS) so no de-dup needed.
- `ScoresFile.read_point/4` checks `.prop`, then `.hrdps.prop`, then
legacy `.ntms` — first hit wins. Cells outside their owning region
read as no-data in the other file's grid, so the order doesn't
matter for correctness.
- `Propagation.warm_cache_and_broadcast/2` reads both files and
merges before broadcasting to the cluster ScoreCache. A single
missing file (HRDPS pre-cycle, HRRR briefly absent) is now OK —
the cache warms with whichever side is available.
Activation:
- runtime.exs cron: HRDPS seed worker fires at HH:35 of each cycle
hour (05:35Z, 11:35Z, 17:35Z, 23:35Z) — 5h after cycle, 30 min
staggered from HRRR's */15 schedule.
- HrdpsGridWorker moduledoc updated: no longer dormant; Rust
prop-grid-rs handles source='hrdps' rows.
- map_live North America bbox extended to (24.5, 60.0, -141.0, -52.0)
for the out-of-coverage banner check; visitors anywhere in CONUS
or the Canadian extent now stay banner-free.
Cleanup:
- mix hrdps.probe deleted — its risks are retired and the production
HrdpsClient covers the same surface.
What's still in motion (not blocking the Canadian /map):
- Per-valid_time profile + scalar artifacts for HRDPS (would let
/weather show Canadian temperature/refractivity layers). Rust
pipeline currently skips those so the HRRR-written companion files
aren't clobbered — separate stage when /weather UX is decided.
- FreshnessMonitor HRDPS staleness tracking — HRDPS has its own cron
and a different cadence than HRRR; HRRR-stale logic doesn't apply
cleanly. Defer until prod tells us what alarms we actually want.
Adds mix hrdps.probe — throwaway wedge that retires URL/projection/decode
risks against MSC Datamart. Verified against five Canadian cities with
plausible 2m temperatures via wgrib2's native rotated-lat/lon support.
Findings update what we knew:
- ECCC reorganized to date-prefixed paths; the old /model_hrdps/ root
404s now. Real URL is dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/...
- HRDPS publishes one variable per GRIB2 file (~3.5 MB each), not
HRRR's bundled wrfsfcf format. ~14 files per cycle/forecast hour.
- wgrib2 -lon LON LAT handles the rotated grid natively — no manual
reprojection math needed.
New plan at 2026-04-29-hrdps-canadian-prop-grid.md supersedes the
2026-04-13 one (which assumed HRRR-style idx + byte-range fetches that
don't apply). Probe deletion is a Stage 10 step in the new plan.
Mix tasks don't run in production, so the orchestration lives in
Microwaveprop.Buildings.BulkFetch — callable from an attached iex on
the prod pod via:
iex> Microwaveprop.Buildings.BulkFetch.run(32.8, -97.0, 500)
Default 500 mi radius around DFW yields ~440 candidate quadkeys at
zoom 9; only those present in the MS dataset index (typically a few
hundred over CONUS) get downloaded. Concurrency 4, per-task timeout
3 min, results logged with downloaded/cached/failed/missing counts.
The mix.task is kept as a dev-only thin wrapper.
Mix.Task.run isn't available in a compiled release, so the in-prod
`bin/microwaveprop eval 'Mix.Tasks.Weather.RebatchAsos.run(...)'`
dispatch crashes. Moves the real logic into a plain module
(`Microwaveprop.Weather.RebatchAsos`) that both the Mix task and a
release eval can call. Uses IO.puts over Mix.shell for the same
reason.
- IemRateLimiter gains AIMD-style adaptive spacing. signal_429/0
widens the current gap (*= 1.5, capped at max_interval_ms default
10s); signal_success/0 narrows it back toward the configured base
(*= 0.92, floored at interval_ms). Self-tunes to IEM's moving
ceiling without needing the manual "safe for 4 pods" constant.
- IemClient now routes every response through a central handle_response
helper that fires the widen/narrow feedback signals, eliminating the
four near-identical case blocks.
- Mix.Tasks.Weather.RebatchAsos collapses any already-queued single-
station "asos" jobs into the batched "asos_batch" shape the
enqueuer now emits, so the pending backfill queue converts to the
new per-request-efficient path instead of draining at the old rate.
Idempotent; supports --dry-run.
2833 tests + credo green.
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Previously the `mix deps.patch` task lived at `lib/mix/tasks/deps.patch.ex`
and the `deps.get` alias invoked it. This broke `mix deps.get --only prod`
in Docker: `lib/` isn't compiled yet when deps are fetched, so the task
module couldn't be resolved and the build failed with
`The task "deps.patch" could not be found`.
Move the patch-application function directly into `mix.exs` as a private
helper and wire it into the alias via `&apply_dep_patches/1`. `mix.exs` is
always loadable, so the function is available the moment `deps.get` runs.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
After the NARR backfill landed, nothing reachable from the app calls
any of these modules. Net removal: ~2700 lines.
Deleted:
- Microwaveprop.Workers.{Era5Fetch,Era5Submit,Era5Poll,Era5MonthBatch}Worker
- Microwaveprop.Weather.{Era5BatchClient,Era5Client}
- Mix.Tasks.Era5Backfill (superseded by BackfillEnqueueWorker cron)
- Matching test files (~1100 LOC of test)
- era5/era5_submit/era5_poll/era5_batch queue blocks from runtime.exs
- era5_req_options Req.Test stub config from test.exs
Kept (actively used or intentionally preserved):
- Era5Profile schema (the era5_profiles table is the NARR target)
- Era5CdsJob schema + its test (inspection handle on the era5_cds_jobs
table, which retains rows from the failed CDS runs)
- :era5 type key in ContactWeatherEnqueueWorker / BackfillEnqueueWorker
(the symbol still means "historical backfill", just routed to NARR now)
Swapped the one remaining live Era5FetchWorker call site in
contact_live/show.ex's maybe_enqueue_era5 path to NarrFetchWorker
(with NarrClient.snap_to_analysis_hour for the 3-hourly slot).
Test suite: 1513 tests, 0 failures (-50 from the deleted era5 test
files). No compile warnings. Credo clean.
The grid worker stopped writing HRRR profiles to the DB on April 14
when the forecast grid moved to /data/scores binary files, but the
24/72h-windowed prune left every is_grid_point=true row older than
72 hours sitting in the table (billions of rows across every
partition back to 2016, ~90% of the 146 GB table size). Nothing
reads them anymore.
Replace prune_old_grid_profiles/0 with purge_grid_point_profiles/0,
which walks every hrrr_profiles partition directly and deletes
is_grid_point=true rows regardless of age. Per-partition DELETEs
avoid a parent-table full scan. QSO-linked rows (is_grid_point=false)
are untouched — those still back contact enrichment and /path.
Call the new purge from the grid worker's end-of-chain hook so any
grid-point row that somehow slips back in gets cleaned up within the
next hour. Add `mix hrrr.purge_grid_points` for a one-shot historical
cleanup.
Full cutover: the propagation_scores Postgres table is gone, and
the binary files under /data/scores are the sole source of truth
for the map render path. Three stacked changes:
1. New migration drops the propagation_scores table and its indexes
(the earlier tuning migrations for it were already applied and
are now no-ops against a missing table, which is fine — Ecto
just runs them on fresh environments).
2. Propagation context is gutted of every GridScore reference.
replace_scores/2 writes files only. upsert_scores/2 is deleted.
load_scores_from_db, available_valid_times_from_db,
point_detail_from_db, point_forecast_from_db, fetch_factors,
coalesce_factors, the Postgres side of prune_old_scores, and
the Postgres fallbacks in latest/earliest_valid_time are all
removed. point_detail always returns an empty factors map now
since factor breakdowns were retired with the table.
3. Deleted modules:
- Propagation.GridScore (the schema)
- Propagation.ScorerDiff (read factors from the table)
- Propagation.AsosNudge (helper for AsosAdjustmentWorker)
- Workers.AsosAdjustmentWorker (its cron was already disabled)
- Mix.Tasks.ScorerDiff (wrapper around the deleted module)
And their tests. AdminTaskWorker's scorer_diff task is a
logging no-op so any queued Oban rows drain cleanly.
Release.scorer_diff stays as a stub that tells the operator.
propagation_prune_worker_test rewritten to exercise ScoresFile
pruning. propagation_test.exs rewritten to use replace_scores +
ScoresFile throughout (no more GridScore / upsert_scores paths).
Aliases: add module aliases for 9 nested module references
Apply: replace apply/3 with direct module attribute calls
Line length: break 1 long spec line
Refactoring: extract helpers to reduce complexity and nesting
in show.ex, radio.ex, weather workers, terrain, duct detection,
backfill dashboard, contact map, and mix tasks
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
- Add `mix backtest --all` for consolidated pass/fail table across all features
- Add Backtest.consolidated_report/2 and to_consolidated_markdown/1
- Add Features.all_features/0 to auto-discover backtestable features
- Add `mix import_contest_logs` for bulk ARRL contest CSV import with dedup
- Fix hrrr_climatology to batch by (month, hour) to avoid query timeout
- Fix Repo.query! result pattern (Postgrex.Result, not tuple)
- Backtest reports for all Phase 1-6 features
Mix tasks that call app.start were also booting Oban's cron scheduler,
causing PropagationGridWorker and other cron jobs to fire during
backfills. Add Oban.pause_all_queues(Oban) immediately after app.start
in every mix task that only needs Repo access.
Phase 3 NEXRAD spike: IEM n0q composite available at 5-min cadence
back to 2022+. Compression-ratio proxy shows afternoon images have
13-81% more texture than dawn (directionally correct), but the n0q
product thresholds out the faint clear-air returns needed for BL
stability detection. Parked until MRMS or Level III products can be
investigated. See docs/research/nexrad_spike.md.
Phase 6: hrrr_climatology table aggregating surface_temp_c by
(lat, lon, month, hour) from the 42M+ hrrr_profiles grid-point
rows. mix hrrr_climatology builds it via a single SQL GROUP BY +
upsert. Backtest.Features.temperature_anomaly computes current_temp
minus climatological mean — the meteorologist's "temperature
deviation above normal" predictor for summer afternoon enhancement.
Duct module (Propagation.Duct):
- refractivity_profile/1: ITU-R P.453 N at each native level
- m_profile/1: modified refractivity M = N + 157*h(km)
- detect_ducts/1: find contiguous regions where dM/dh < 0, returning
base/top height, thickness, and M-deficit per duct
- min_trapped_frequency_ghz/1: waveguide approximation (Bean & Dutton)
for the minimum frequency a duct of given geometry can trap
- analyze/1: full pipeline from native profile to duct list + best
trapped frequency across all ducts
Derive task updated to also compute ducts JSONB and best_duct_band_ghz
alongside the Phase 2 turbulence fields.
Backtest features: duct_thickness, best_duct_freq, duct_usable_10ghz,
duct_usable_24ghz, duct_usable_47ghz.
Real-data validation: 2022-08-20 12Z TX profile shows 0 ducts (M
increases monotonically) — correct for a well-mixed boundary layer
on a turbulent August afternoon (Ri=0.16).
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
mix backtest --feature NaiveGradient now resolves to the same function
as --feature naive_gradient. Names are normalized via Macro.underscore,
so any casing works. Also prints the available functions when an
unknown feature is requested so typos don't produce an opaque
UndefinedFunctionError deep in the Enum.map stack.
Add Microwaveprop.Backtest: a feature-evaluation framework that runs
a (lat, lon, valid_time) -> float function over the historical QSO
corpus and a matched random-time baseline, reporting distribution
statistics, distance-binned lift, and band-stratified lift.
Adds four baseline feature wrappers around the current scorer inputs
(naive_gradient, td_depression, time_of_day, pressure), a mix backtest
CLI, and the first set of baseline reports under priv/backtest_reports
so downstream phases have a frozen reference point to compare against.
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.
- Era5BatchClient builds the monthly CDS requests, extracts every
(lat, lon, hour) from the GRIB2 blob with wgrib2, derives
refractivity params, and bulk-inserts in 2k-row chunks with
on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
→ enqueue the month-batch for the point's tile-month and return.
No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.
Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
Enqueues ERA5 fetch jobs for contacts where HRRR is unavailable.
Deduplicates by rounded grid point and hour, skips existing profiles.
Usage: mix era5_backfill [limit]
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
- Cache raw GRIB2 byte ranges to ~/hrrr in dev (never deleted)
- mix hrrr_backfill re-fetches QSO-linked HRRR profiles with 13 levels
- Supports --all (re-fetch everything) and --limit N flags
- Groups by HRRR hour to batch requests, 500ms rate limit
- AWS archive has full HRRR history, same URL pattern as live feed
- Nx, Axon, EXLA, Polaris deps restricted to only: [:dev, :test]
- model.ex and training mix tasks moved to lib_ml/ (compiled via
elixirc_paths in dev/test only)
- load_ml_model uses Code.ensure_loaded? + apply/3 to avoid
compile-time references to ML modules in production
- Verified: MIX_ENV=prod compiles clean with no ML warnings
Rename all modules, functions, variables, routes, and UI text from
qso/qsos to contact/contacts. Database table stays as "qsos" to avoid
migration. Add /qsos -> /contacts redirects for old URLs.
Add SFI, Kp max (solar), K-index, lifted index (sounding stability),
and ducting_detected (HRRR) as model features. Training now joins to
solar_indices and nearest sounding (within 6 hours) for both phases.
Model can learn solar/geomagnetic effects if they exist in the data.
- 15 features: add surface_refractivity and latitude
- Bigger network: 128→64→32 (3 hidden layers)
- Phase 1: pretrain on 500K stratified algorithm scores (all seasons/locations)
- Phase 2: fine-tune on 57K real QSO-HRRR matched data (percentile target)
- Lower LR (0.0003) for fine-tuning to preserve pretrained knowledge
- Model.train accepts :initial_state option for transfer learning
- 3 hidden layers instead of 2 for better feature interaction learning
- Target is within-band distance percentile (0-1) instead of raw
normalized distance — reduces noise from operator/equipment variation
Features: HRRR conditions averaged at both QSO endpoints + solar time
Target: distance_km normalized per-band (distance / p99_range, capped at 1.0)
This trains on actual propagation outcomes from 57K+ QSOs, not the
hand-tuned algorithm output.
Raw features had vastly different scales (pressure ~1013, sin/cos ~[-1,1])
causing gradient explosion. Normalize all atmospheric features to ~[0,1]
using known physical bounds. Add Polaris dep for optimizer.
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2)
- PropagationGridWorker fetches all 19 forecast hours per run
- Propagation.scores_at/3 queries scores at specific valid_time
- Propagation.available_valid_times/1 returns all forecast times for timeline
- Pruning keeps scores with valid_time >= now - 2h (forecast-aware)
- MapLive: select_time event, timeline data pushed to JS
- JS: forecast timeline bar at bottom of map with clickable hour buttons
- PubSub broadcast sends list of valid_times instead of single time
