Replaces the "tropo composite + post-hoc aurora boost" model with a
max-over-mechanisms dispatcher so each band's score reflects the
dominant propagation channel rather than summing alternative paths.
- BandConfig.mechanisms/1 derives the stack from frequency:
≤432 MHz gets [:tropo, :aurora]; >432 MHz stays [:tropo]. Bands
may override with an explicit :mechanisms field.
- Scorer.mechanism_score/3 returns a standalone integer score for
one mechanism (:tropo delegates to composite_score; :aurora is a
clean Kp/freq function, not a boost on top of tropo).
- Scorer.cell_score/2 takes max across the band's mechanisms so a
Kp-6 storm scores 6m at 85 even when the tropo picture is poor,
instead of conflating the two channels.
Es and F2 stay in PathCompute since they're path-distance-dependent.
The legacy aurora_boost path is unchanged so no callers shift today.
`Scorer.aurora_boost(score, kp, band_config)` is a terminal boost
applied after `composite_score` in the same shape as the existing
`commercial_link_boost`. It tracks the NOAA G-scale (Kp 5+ opens
auroral E-region paths up to ~2500 km) and is freq-gated to
≤ 432 MHz — auroral propagation is observed almost exclusively on
50/144/222, occasionally on 432, and never at microwave.
Boost magnitudes:
* Kp < 4 (quiet) — pass-through
* Kp 4 — +5
* Kp 5 (G1) — +15
* Kp 6 (G2) — +25
* Kp ≥ 7 (G3+) — +35, clamped to 100
Wired into `Propagation.score_grid_point/4` via the existing
`hrrr_profile` map: callers populate `:kp_index` once per cycle so
the per-grid-point band loop never queries the DB. The UI fallback
path (`factors_from_profile/5`) sources Kp from
`SpaceWeather.latest_kp/0` directly. Other call sites
(GEFS forecast, path-compute, contact-show) can opt in by adding
`:kp_index` to their profile map.
Diagnostic factor `:kp_index` is stashed on VHF/low-UHF results so
the operator-facing factor breakdown can show why a band scored
above its quiet-conditions baseline. Microwave bands don't carry
the diagnostic — they're never affected by the boost.
Refactored `score_with_algorithm/7` to delegate band-level
finalization to `finalize_band_result/5`, keeping the parent
function under credo's cyclomatic-complexity limit.
NOTE: this only affects the Elixir scoring path — the live grid
served by `prop_grid_rs` (Rust) does its own scoring and does not
yet honor Kp. Adding aurora to the Rust side is follow-up work.
- Unify haversine_km / deg_to_rad: Geo is the canonical home (atan2
form for antipodal stability); Radio, common_volume, rain_scatter,
ionosphere, terrain/srtm, terrain/elevation_client now delegate.
- Drop safe_min/safe_max wrappers in favor of Enum.min/max defaults.
- Move parse_float / parse_int to MicrowavepropWeb.LiveHelpers; eme,
path, and rover LiveViews import from there.
- Extract MicrowavepropWeb.LocationResolver from the eme/path
resolve_source / resolve_location duplicate. Standardize the kind
key and "Invalid grid square" error message.
- Convert Scorer cond chains (humidity, td_depression, sky, wind,
rain, pwat, pressure) to multi-clause guarded defs, matching the
existing classify_time_period style.
- extract_profile_fields uses a named map accumulator instead of a
6-tuple; build_path_conditions guards on %{temps: []} / %{dewpoints: []}.
- Collapse scores_file clamp/3 to max |> min.
- humidity_effect_label lives in BandConfig; map_live and path_live
both use it.
Both retired in light of the 2026-04-25 revision report
(`docs/algo-reports/2026-04-25-algo-revisions.md`):
* HPBL boundary-layer multiplier in `Scorer.score_refractivity/{3,4,5}`
is gone. On the n=47,418 10 GHz HRRR-matched corpus rho_hpbl is
+0.004; binned distance is flat to within ±5 % across 200–2,000 m.
The previously reported 2.3× ratio between shallow and deep HPBL
bins was a small-corpus artefact that disappeared once the matched
corpus passed ~5,000 contacts.
* Native-profile 1.15× duct boost is gone. At 10 GHz on n=52,341
matched contacts, cells where `best_duct_band_ghz` ≥ band ran
*shorter* than no-duct cells (198 km vs 211 km, n=173 vs 44,658).
The Bulk Richardson gate that existed only to suppress that boost
in turbulent conditions is also retired; arity preserved on every
`score_refractivity` overload so callers compile unchanged.
Same retirements applied in the Rust port (`rust/prop_grid_rs/src/scorer.rs`)
to keep the Elixir/Rust parity tests green.
algo.md updated end-to-end:
* Finding 4 (HPBL) rewritten to "no usable signal" with the
n=47,418 bin table.
* Finding 5 (gradient) tightened: load-bearing only at 24 GHz,
set band-specific gradient weight to 0 outside [10, 47] GHz on
the next derive_band_weights run.
* Finding 8 (time-of-day) augmented with the robust hour-bucketed
amplitude index (40 / 82 / 101 % at 10 / 24 / 47 GHz) and a
selection-bias caveat for VHF/UHF (where 129–148 % amplitude
reflects evening contest scheduling, not propagation).
* Finding 9 (sounding) refreshed against the n=27,058 corpus.
* Part 2c "Native-profile duct boost" rewritten as
"retired 2026-04-25" with the falsifying join.
* Part 2c "Commercial-link inverse sensor" promoted from deferred
with the 22-day, 38k-sample SNMP corpus, the +0.61 / −0.61 PWAT
and pressure correlations against the 37-hour HRRR overlap, and
the 04–05 / 10–12 local two-peak morning fade window.
* Part 2b "Pressure" annotated with the U-shape at 10 GHz and the
next-iteration target.
* Part 2d "What stayed, what left" + "Open items" updated to
reflect the four code/doc moves.
119 Elixir scorer tests green, 134 Rust scorer tests green, 2,888
total Elixir tests + 221 properties green via `mix precommit`.
- endpoint.ex log_level/1 now filters by conn.request_path instead of
conn.path_info. Plug.Router.forward/2 (deps/plug/lib/plug.ex:170)
rewrites path_info to the unmatched remainder and extends script_name
with the matched prefix before dispatching to the forwarded plug.
/metrics is routed via forward "/metrics", MetricsPlug; by the time
Plug.Telemetry's before_send callback fires inside MetricsPlug.call/2
the conn it observes has path_info: [] and script_name: ["metrics"],
so the old log_level(%{path_info: ["metrics" | _]}) clause never
matched and the default :info level fired. request_path is set by
the adapter at entry and is never rewritten, making it the correct
discriminator. New regression test in metrics_log_suppression_test.exs
captures both the direct shape and the integration path via Plug.Test.
- Scorer.dbz_to_rain_rate_mmhr/1 now uses a compile-time @mp_inv_b
constant (1 / 1.6) instead of dividing on every rain pixel.
composite_score/2's band-invariant fallback switched from four
separate short-circuits (which silently
mixed cached and freshly-computed values if only some keys were
passed) to a single Map.has_key?/2 branch that honors the "all or
none" contract. Dropped unused band_invariant_tod/1 helper.
- Propagation.replace_scores span scope tightened: the
Instrument.span([:db, :replace_scores]) now wraps only the per-band
ScoresFile.write! loop, not the upstream Enum.group_by grouping
phase. The span name + metadata stay unchanged so Grafana panels
keep working, but the fixed telemetry dispatch cost (~100µs x 2)
is no longer paid for trivially small result sets.
- Added @type t :: %__MODULE__{...} to Accounts.UserToken and
Weather.HrrrClimatology — the last two schemas that lacked one.
Elixir 1.19's set-theoretic inference benefits from every struct
having an explicit t/0 so callers can flow through tightly.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 3 pre-existing flakes, 0 new regressions.
Two wins on the hourly propagation pipeline:
1. Parallelize the chain. seed_chain was enqueuing only f00 and
each step self-enqueued f00+1, serializing ~2.5 min × 19
forecast hours into a ~48 min chain. Fan out all 19 jobs at
once — they're genuinely independent (different HRRR URLs,
different output files) — and let queue concurrency (2 slots
× 3 pods = 6 parallel workers) drop wall time to ~10 min.
Side effect: one permanently-failing step no longer takes out
the rest of the chain, so the rescue-chain logic I added last
commit becomes unnecessary. Removed.
The :final cleanup (retain_window + purge) used to run on the
fh=18 transition; with parallel execution we don't know which
step finishes last. Cleanup now relies on the existing
PropagationPruneWorker 15-min cron for time-based pruning.
Stale chain leftovers are already a minor concern with the
15-min prune; if file bloat becomes real, PruneWorker can
gain retain_window later.
2. Hoist band-invariant factors. score_time_of_day, score_sky,
score_wind, score_pressure depend on conditions only, not the
band — but composite_score was recomputing them 17 times per
point. Added Scorer.precompute_band_invariants/1, called once
per point before the bands loop; composite_score uses the
cached values when present and falls back to computing for
any caller that doesn't pre-warm (test harness, path
integrator). Saves ~30% of the scoring inner loop.
Added 5-arity Scorer.score_refractivity/5 that takes bulk_richardson
alongside best_duct_band_ghz. The existing 1.15× boost now only
applies when Richardson is in the stable regime (< 25) — native-
profile duct cells average 8.7-18.4 for ducting vs 38.3 for
non-ducting, so a duct-band reading under turbulent conditions is
more likely to be torn up by mechanical mixing than to carry a
beyond-LOS signal.
Wiring:
- Weather.nearest_native_duct_info/3 returns {ghz, richardson}; the
bare nearest_native_duct_ghz/3 now delegates.
- PathLive and Propagation.score_with_algorithm/7 both pull the
Richardson value into the conditions map alongside best_duct_band.
- Scorer.composite_score/2 reads conditions[:bulk_richardson] and
passes it into score_refractivity/5.
- Backward compat: 4-arity variant and nil Richardson keep the old
unconditional-boost behaviour so existing callers don't regress.
Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).
- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
- Replace %Struct{} with Struct.t() in all @spec annotations
- Replace length(x) > 0 with x != [] in test assertions
- Fix multi-line spec struct references in weather.ex
- Stream profile storage and score upsert instead of materializing
full 20k+ item lists (propagation_grid_worker, propagation.ex)
- GC between forecast hours and store/compute phases to reclaim
~400 MB of grid data between steps
- Single-pass field extraction in scorer.ex path_integrated_conditions
instead of 6 separate Enum traversals
- Eliminate intermediate merged map in fetch_grid by combining
merge + profile build into one pipe
- Fix UUID bug: bingenerate → generate in native grid worker
(same issue previously fixed in nexrad_worker)
Add Propagation.Region module with 8 CONUS climate zones (gulf_coast,
southeast, southern_plains, corn_belt, northeast, desert_southwest,
pacific_northwest, mountain_west) and per-region monthly seasonal
adjustment multipliers.
The scorer's score_season now takes lat/lon and applies a regional
multiplier from Region.seasonal_adjustment on top of the band's
seasonal_base + seasonal_adj. Gulf coast August gets a 1.15x boost
(drier, better for ducting) while Corn Belt August gets a 0.80x
penalty (corn evapotranspiration = miserable dewpoints).
Adjustments are hand-tuned starting points from the meteorologist's
qualitative guidance. Phase 9 recalibration will refine them from
backtest data.
Contact scoring now uses all HRRR profiles along the path instead of
just pos1. Aggregation strategy:
- Best along path for beneficial factors (refractivity, pressure)
- Worst along path for harmful factors (rain, wind)
- Average for neutral factors (temp, dewpoint, PWAT, BL depth)
Scorer.path_integrated_conditions/2 merges multiple profiles into a
single conditions map. Falls back gracefully to single-profile scoring
when only one profile is available.
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
Score time-of-day per grid point using longitude/15 solar offset instead of
hardcoded CST/CDT. Add PWAT as 10th scoring factor. Refine pressure thresholds.
Update ML model and training pipeline to use local solar time.
