Wire NEXRAD, native ducts, and commercial links into scoring

Three signal sources we already collect but weren't using:

* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
  as max of HRRR-derived and NEXRAD-derived rate so fast convective
  cells between HRRR hourly analyses can still trigger the rain penalty.
  Only active on f00 — forecast hours can't see future radar. New
  Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
  hail-safe ceiling.

* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
  applies a 1.15× boost when the cell's native-resolution duct supports
  the target band's frequency. HRRR pressure-level gradients
  systematically under-read thin trapping layers the native profile can
  resolve. Sub-band ducts do NOT boost — they're evidence that the
  gradient we have is all there is at the target frequency.

* Commercial LOS link rx_power fading → inverse tropo sensor.
  Commercial.link_degradation_at/3 computes the average 7-day-baseline
  vs current delta across enabled links within 75 km, ignoring links
  where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
  to the composite score for 3+ dB of fading. ~150 km radius around
  DFW is the only zone this helps today, but it's the first *measured*
  signal in the algorithm vs the model-derived proxies.

Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.

1,359 tests, 0 failures.
This commit is contained in:
Graham McIntire 2026-04-13 12:08:15 -05:00
parent bc6377feb3
commit 86364f43aa
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
9 changed files with 693 additions and 8 deletions

38
algo.md
View file

@ -766,12 +766,46 @@ Per the user's directive to model **all bands ≥902 MHz**, these are added to `
### Open items / next analyses ### Open items / next analyses
1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny. 1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny.
2. **NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring. 2. ~~**NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring.~~ **Landed.** See "NEXRAD composite reflectivity" below.
3. **`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude. 3. ~~**`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude.~~ **Landed.** See "Native-profile duct boost" below.
4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band. 4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band.
A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL. A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL.
### NEXRAD composite reflectivity → rain-attenuation score
HRRR hourly precipitation accumulation lags fast-moving convective cells by up to 59 minutes. NEXRAD n0q composite reflectivity runs at 5-minute cadence and catches those cells at the moment they pass over a grid cell. The scoring pipeline now takes the *max* of the HRRR-derived rain rate (from `precip_mm`) and the NEXRAD-derived rain rate (from `max_reflectivity_dbz` via Marshall-Palmer) so either source can trigger the rain penalty without double-counting.
`Scorer.dbz_to_rain_rate_mmhr/1` implements:
```
R (mm/hr) = (Z / 200)^(1/1.6) where Z = 10^(dBZ/10)
```
with a 5 dBZ noise floor (ground clutter / clear air) and a 150 mm/hr ceiling (hail contamination). Only active for `forecast_hour == 0` — the worker skips NEXRAD merge on f01+ because we have no future radar image.
### Native-profile duct boost
The base refractivity score uses HRRR's 13 pressure levels, which systematically under-read thin trapping layers (see Part 2c — only 0.12% of native-resolution cells support ≥15 GHz, but the pressure-level data looks the same above and below that threshold). When `hrrr_native_profiles.best_duct_band_ghz` is present and its value is ≥ the target band's frequency, `Scorer.score_refractivity/4` multiplies the base score by 1.15×. A duct that only supports sub-band frequencies does *not* boost — it's evidence that the gradient we have is all there is at the target band.
This is an additive correction to the HPBL multiplier introduced earlier in Part 2c. The two multipliers compose: a thin shallow-BL cell with a 24-GHz-supporting native duct gets the full 1.051.10× HPBL bonus *and* the 1.15× native-duct bonus, clamped at 100.
### Commercial-link inverse sensor
Seven af11x / af60 commercial microwave links around Princeton TX (33.2°N, 96.5°W) are polled every 5 minutes via SNMP and stored in `commercial_samples`. These are short (26 km) LOS paths where the same refractivity anomaly that helps beyond-LOS amateur propagation degrades the link's rx_power through multipath fading.
`Commercial.link_degradation_at/3` computes the average of (7-day baseline rx_power current rx_power) across all healthy (`link_state == 1`) links within a configurable radius of a target point (default 75 km). The worker calls this for every grid cell on `forecast_hour == 0` runs; commercial links only exist near DFW so almost every cell gets `nil` at near-zero cost.
`Scorer.commercial_link_boost/2` then adds a terminal boost to the composite score:
| degradation_db | Bonus | Interpretation |
|---|---|---|
| <3 dB | 0 | Noise floor |
| 38 dB | +2 to +10 | Mild multipath — weak enhancement |
| ≥8 dB | +10 to +25 (clamped ≤100) | Deep fade — strong ducting signature |
This is the first *measured* signal in the algorithm — every other factor is a model-derived proxy. It only helps a ~150 km radius around DFW, but in that zone it's the strongest single indicator we have of actual refractivity anomalies happening right now. Out-of-zone cells see no change.
--- ---
## Part 3: Band Configuration ## Part 3: Band Configuration

View file

@ -7,6 +7,11 @@ defmodule Microwaveprop.Commercial do
alias Microwaveprop.Commercial.Sample alias Microwaveprop.Commercial.Sample
alias Microwaveprop.Repo alias Microwaveprop.Repo
@default_radius_km 75.0
@default_baseline_days 7
@default_current_window_seconds 900
@min_baseline_samples 5
@spec enabled_links() :: [Link.t()] @spec enabled_links() :: [Link.t()]
def enabled_links do def enabled_links do
Link Link
@ -14,6 +19,142 @@ defmodule Microwaveprop.Commercial do
|> Repo.all() |> Repo.all()
end end
@doc """
Returns the aggregate commercial-link rx_power degradation at a location,
or `nil` if no healthy links are in range.
Commercial LOS microwave links near the target point act as an inverse
tropo sensor: when their rx_power drops significantly below the 7-day
baseline without an obvious equipment cause (link_state still 1), the
refractivity structure that hurts their short Fresnel zone is usually
the same structure that *helps* long beyond-LOS amateur propagation.
Result map:
%{
degradation_db: float (positive means worse than baseline),
baseline_dbm: float,
current_dbm: float,
n_links: integer
}
## Options
* `:radius_km` distance from the target point to include a link (default 75)
* `:baseline_days` days of history for the baseline average (default 7)
* `:current_window_seconds` how recent "current" rx must be (default 900)
"""
@spec link_degradation_at({float(), float()}, DateTime.t(), keyword()) :: map() | nil
def link_degradation_at({lat, lon}, valid_time, opts \\ []) do
radius_km = Keyword.get(opts, :radius_km, @default_radius_km)
baseline_days = Keyword.get(opts, :baseline_days, @default_baseline_days)
current_window = Keyword.get(opts, :current_window_seconds, @default_current_window_seconds)
candidates =
enabled_links()
|> Enum.map(fn link -> {link, link_endpoint(link)} end)
|> Enum.reject(fn {_link, endpoint} -> is_nil(endpoint) end)
|> Enum.filter(fn {_link, {elat, elon}} ->
haversine_km(lat, lon, elat, elon) <= radius_km
end)
baseline_cutoff = DateTime.add(valid_time, -baseline_days * 24 * 3600, :second)
current_cutoff = DateTime.add(valid_time, -current_window, :second)
results =
candidates
|> Enum.map(fn {link, _endpoint} ->
link_degradation(link.id, baseline_cutoff, current_cutoff, valid_time)
end)
|> Enum.reject(&is_nil/1)
case results do
[] ->
nil
list ->
baseline_avg = average(Enum.map(list, & &1.baseline_dbm))
current_avg = average(Enum.map(list, & &1.current_dbm))
%{
degradation_db: Float.round(baseline_avg - current_avg, 2),
baseline_dbm: Float.round(baseline_avg, 2),
current_dbm: Float.round(current_avg, 2),
n_links: length(list)
}
end
end
defp link_degradation(link_id, baseline_cutoff, current_cutoff, valid_time) do
baseline =
Sample
|> where([s], s.link_id == ^link_id)
|> where([s], s.sampled_at >= ^baseline_cutoff and s.sampled_at < ^current_cutoff)
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|> select([s], avg(s.rx_power_0))
|> Repo.one()
{current, baseline_n} = fetch_current_and_baseline_count(link_id, current_cutoff, valid_time)
cond do
is_nil(baseline) or is_nil(current) ->
nil
baseline_n < @min_baseline_samples ->
nil
true ->
%{baseline_dbm: to_float(baseline), current_dbm: current}
end
end
defp to_float(%Decimal{} = d), do: Decimal.to_float(d)
defp to_float(n) when is_number(n), do: n * 1.0
defp fetch_current_and_baseline_count(link_id, current_cutoff, valid_time) do
current =
Sample
|> where([s], s.link_id == ^link_id)
|> where([s], s.sampled_at >= ^current_cutoff and s.sampled_at <= ^valid_time)
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|> order_by([s], desc: s.sampled_at)
|> limit(1)
|> select([s], s.rx_power_0)
|> Repo.one()
baseline_n =
Sample
|> where([s], s.link_id == ^link_id)
|> where([s], s.sampled_at < ^current_cutoff)
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|> select([s], count(s.id))
|> Repo.one()
{current, baseline_n}
end
defp link_endpoint(%Link{endpoint_a: %{"lat" => lat, "lon" => lon}}) when is_number(lat) and is_number(lon),
do: {lat * 1.0, lon * 1.0}
defp link_endpoint(_), do: nil
defp haversine_km(lat1, lon1, lat2, lon2) do
r = 6371.0
phi1 = lat1 * :math.pi() / 180
phi2 = lat2 * :math.pi() / 180
dphi = (lat2 - lat1) * :math.pi() / 180
dlambda = (lon2 - lon1) * :math.pi() / 180
a =
:math.sin(dphi / 2) * :math.sin(dphi / 2) +
:math.cos(phi1) * :math.cos(phi2) * :math.sin(dlambda / 2) * :math.sin(dlambda / 2)
c = 2 * :math.atan2(:math.sqrt(a), :math.sqrt(1 - a))
r * c
end
defp average([]), do: 0.0
defp average(list), do: Enum.sum(list) / length(list)
@spec create_link(map()) :: {:ok, Link.t()} | {:error, Ecto.Changeset.t()} @spec create_link(map()) :: {:ok, Link.t()} | {:error, Ecto.Changeset.t()}
def create_link(attrs) do def create_link(attrs) do
%Link{} %Link{}

View file

@ -90,10 +90,11 @@ defmodule Microwaveprop.Propagation do
longitude: longitude, longitude: longitude,
pressure_mb: hrrr_profile.surface_pressure_mb, pressure_mb: hrrr_profile.surface_pressure_mb,
prev_pressure_mb: nil, prev_pressure_mb: nil,
rain_rate_mmhr: Scorer.precip_to_rate_mmhr(hrrr_profile[:precip_mm]), rain_rate_mmhr: merged_rain_rate(hrrr_profile),
min_refractivity_gradient: hrrr_profile[:native_min_gradient] || derived[:min_refractivity_gradient], min_refractivity_gradient: hrrr_profile[:native_min_gradient] || derived[:min_refractivity_gradient],
bl_depth_m: hrrr_profile[:hpbl_m], bl_depth_m: hrrr_profile[:hpbl_m],
pwat_mm: hrrr_profile[:pwat_mm] pwat_mm: hrrr_profile[:pwat_mm],
best_duct_band_ghz: hrrr_profile[:best_duct_freq_ghz] || hrrr_profile[:best_duct_band_ghz]
} }
duct_info = duct_info =
@ -106,13 +107,45 @@ defmodule Microwaveprop.Propagation do
} }
end end
link_degradation = hrrr_profile[:commercial_link_degradation]
Enum.map(BandConfig.all_bands(), fn band_config -> Enum.map(BandConfig.all_bands(), fn band_config ->
result = Scorer.composite_score(conditions, band_config) result = Scorer.composite_score(conditions, band_config)
result = Map.put(result, :band_mhz, band_config.freq_mhz)
boosted_score =
if link_degradation do
Scorer.commercial_link_boost(result.score, link_degradation)
else
result.score
end
result =
result
|> Map.put(:score, boosted_score)
|> Map.put(:band_mhz, band_config.freq_mhz)
result =
if link_degradation do
put_in(result, [:factors, :commercial_link_degradation], link_degradation)
else
result
end
if duct_info, do: put_in(result, [:factors, :duct_info], duct_info), else: result if duct_info, do: put_in(result, [:factors, :duct_info], duct_info), else: result
end) end)
end end
# Pick the heavier of HRRR's hourly accumulation-derived rate and NEXRAD's
# reflectivity-derived rate. NEXRAD catches fast-moving convective cells that
# fall between HRRR hourly analyses; HRRR catches broad stratiform rain that
# NEXRAD reports as low dBZ. Taking max lets either source trigger the rain
# penalty without double-counting.
defp merged_rain_rate(hrrr_profile) do
hrrr_rate = Scorer.precip_to_rate_mmhr(hrrr_profile[:precip_mm])
nexrad_rate = Scorer.dbz_to_rain_rate_mmhr(hrrr_profile[:nexrad_max_reflectivity_dbz])
max(hrrr_rate, nexrad_rate)
end
@doc """ @doc """
Upsert propagation scores in batches within a transaction so readers see all-or-nothing. Upsert propagation scores in batches within a transaction so readers see all-or-nothing.

View file

@ -50,6 +50,27 @@ defmodule Microwaveprop.Propagation.Scorer do
def precip_to_rate_mmhr(mm) when mm > 0, do: mm / 1 def precip_to_rate_mmhr(mm) when mm > 0, do: mm / 1
def precip_to_rate_mmhr(_mm), do: 0.0 def precip_to_rate_mmhr(_mm), do: 0.0
@doc """
Converts NEXRAD composite reflectivity (dBZ) to rain rate (mm/hr) via the
Marshall-Palmer Z-R relationship `Z = 200 * R^1.6`, so `R = (Z/200)^(1/1.6)`
with `Z = 10^(dBZ/10)`.
Clipping rules:
* Below 5 dBZ not rain (ground clutter / clear air). Returns 0.0.
* Above 150 mm/hr hail contamination (55+ dBZ can return >100 mm/hr
from pure Marshall-Palmer). The ITU-R P.838 rain table tops out at
~150 mm/hr so we clip there to stay inside the calibrated regime.
"""
@spec dbz_to_rain_rate_mmhr(number() | nil) :: float()
def dbz_to_rain_rate_mmhr(nil), do: 0.0
def dbz_to_rain_rate_mmhr(dbz) when dbz < 5.0, do: 0.0
def dbz_to_rain_rate_mmhr(dbz) do
z = :math.pow(10, dbz / 10)
r = :math.pow(z / 200, 1 / 1.6)
min(r, 150.0)
end
# ── Factor 1: Humidity ──────────────────────────────────────────── # ── Factor 1: Humidity ────────────────────────────────────────────
@doc """ @doc """
@ -163,9 +184,27 @@ defmodule Microwaveprop.Propagation.Scorer do
applied to *both* the threshold-matched score and the default fallback. applied to *both* the threshold-matched score and the default fallback.
""" """
@spec score_refractivity(number() | nil, number() | nil, map()) :: integer() @spec score_refractivity(number() | nil, number() | nil, map()) :: integer()
def score_refractivity(nil, _bl_depth_m, _band_config), do: 50 def score_refractivity(min_gradient, bl_depth_m, band_config) do
score_refractivity(min_gradient, bl_depth_m, nil, band_config)
end
def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do @doc """
Scores refractivity with optional native-profile duct info.
`best_duct_band_ghz` comes from `hrrr_native_profiles` and represents the
highest frequency the cell's native-resolution duct can trap. When it's
the target band's frequency the base score is boosted 1.15× because
HRRR pressure-level gradients systematically under-read thin ducts the
native profile can resolve (see Part 2c Apr 13 2026 findings).
A duct that only supports lower frequencies does NOT boost the score
it's a signal that the gradient we have is *all there is* at the target
band.
"""
@spec score_refractivity(number() | nil, number() | nil, number() | nil, map()) :: integer()
def score_refractivity(nil, _bl_depth_m, _best_duct_band_ghz, _band_config), do: 50
def score_refractivity(min_gradient, bl_depth_m, best_duct_band_ghz, %{freq_mhz: freq_mhz, humidity_effect: effect}) do
thresholds = BandConfig.refractivity_thresholds() thresholds = BandConfig.refractivity_thresholds()
base = base =
@ -178,7 +217,9 @@ defmodule Microwaveprop.Propagation.Scorer do
if effect == :beneficial, do: beneficial_default, else: harmful_default if effect == :beneficial, do: beneficial_default, else: harmful_default
end end
apply_hpbl_multiplier(base, bl_depth_m) base
|> apply_hpbl_multiplier(bl_depth_m)
|> apply_native_duct_boost(best_duct_band_ghz, freq_mhz)
end end
# HPBL multiplier — applied to the base refractivity score. Calibrated to the # HPBL multiplier — applied to the base refractivity score. Calibrated to the
@ -192,6 +233,60 @@ defmodule Microwaveprop.Propagation.Scorer do
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92) defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92)
defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78) defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78)
# Native-profile duct boost — 1.15× when the cell's best duct supports the
# target band's frequency. HRRR pressure-level gradients systematically
# under-read thin ducts (~250m vertical spacing vs 50 native hybrid levels),
# so when hrrr_native_profiles reports a duct band at or above the target
# frequency, the base gradient score is under-estimating the real channel.
defp apply_native_duct_boost(base, nil, _freq_mhz), do: base
defp apply_native_duct_boost(base, best_duct_band_ghz, freq_mhz) do
target_ghz = freq_mhz / 1_000
if best_duct_band_ghz >= target_ghz do
clamp_score(base * 1.15)
else
base
end
end
@doc """
Inverse-sensor boost from commercial LOS link degradation.
Nearby short-path commercial links (11/24/68 GHz in the DFW area) act as a
physical refractivity anomaly sensor: when their rx_power drops 5+ dB below
the 7-day baseline *without* an equipment cause, the same multipath fading
that hurts their short Fresnel zone is what carries the beyond-LOS paths.
This is a terminal boost applied once per composite score, not per-band
the underlying physics is band-agnostic at the resolution we can measure.
Takes the incoming composite (or any 0-100) score and a degradation map
from `Commercial.link_degradation_at/3`; returns a boosted integer score.
* nil or empty result (`n_links == 0`) no-op
* <3 dB noise floor, no-op
* 38 dB mild boost (+0 to +10)
* 8 dB strong boost (+10 to +25, clamped 100)
"""
@spec commercial_link_boost(number(), map() | nil) :: integer()
def commercial_link_boost(score, nil), do: clamp_score(score)
def commercial_link_boost(score, %{n_links: 0}), do: clamp_score(score)
def commercial_link_boost(score, %{degradation_db: db}) when db < 3.0, do: clamp_score(score)
def commercial_link_boost(score, %{degradation_db: db}) when db < 8.0 do
# 3 dB → +2; 7.9 dB → ~+10 (linear interpolation)
bonus = 2 + (db - 3.0) * 8.0 / 5.0
clamp_score(score + bonus)
end
def commercial_link_boost(score, %{degradation_db: db}) do
# 8 dB → +10; 16 dB → +25 (linear)
bonus = min(25.0, 10 + (db - 8.0) * 15.0 / 8.0)
clamp_score(score + bonus)
end
defp clamp_score(value) do defp clamp_score(value) do
value |> round() |> max(0) |> min(100) value |> round() |> max(0) |> min(100)
end end
@ -381,6 +476,7 @@ defmodule Microwaveprop.Propagation.Scorer do
score_refractivity( score_refractivity(
conditions.min_refractivity_gradient, conditions.min_refractivity_gradient,
conditions.bl_depth_m, conditions.bl_depth_m,
conditions[:best_duct_band_ghz],
band_config band_config
), ),
sky: score_sky(conditions.sky_cover_pct), sky: score_sky(conditions.sky_cover_pct),

View file

@ -12,6 +12,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
max_attempts: 3, max_attempts: 3,
unique: [period: 10_800, states: [:available, :scheduled, :executing, :retryable]] unique: [period: 10_800, states: [:available, :scheduled, :executing, :retryable]]
alias Microwaveprop.Commercial
alias Microwaveprop.Propagation alias Microwaveprop.Propagation
alias Microwaveprop.Propagation.BandConfig alias Microwaveprop.Propagation.BandConfig
alias Microwaveprop.Propagation.Grid alias Microwaveprop.Propagation.Grid
@ -20,6 +21,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
alias Microwaveprop.Weather.GridCache alias Microwaveprop.Weather.GridCache
alias Microwaveprop.Weather.HrrrClient alias Microwaveprop.Weather.HrrrClient
alias Microwaveprop.Weather.HrrrNativeClient alias Microwaveprop.Weather.HrrrNativeClient
alias Microwaveprop.Weather.NexradClient
alias Microwaveprop.Weather.SoundingParams alias Microwaveprop.Weather.SoundingParams
require Logger require Logger
@ -93,6 +95,27 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
grid_data = merge_native_duct_data(grid_data, run_time, forecast_hour) grid_data = merge_native_duct_data(grid_data, run_time, forecast_hour)
:erlang.garbage_collect() :erlang.garbage_collect()
# NEXRAD current-hour composite reflectivity catches fast-moving
# convective cells between HRRR hourly analyses. Only useful for
# f00 — forecast hours can't see the future radar image.
grid_data =
if forecast_hour == 0 do
merge_nexrad_data(grid_data, valid_time)
else
grid_data
end
# Commercial-link inverse sensor — only meaningful for f00 because
# the measurement is of the current atmospheric state, not a forecast.
grid_data =
if forecast_hour == 0 do
merge_commercial_link_data(grid_data, valid_time)
else
grid_data
end
:erlang.garbage_collect()
# Build weather cache rows from in-memory grid_data — avoids the ~20s # Build weather cache rows from in-memory grid_data — avoids the ~20s
# JSONB round trip that was crashing the worker before f01f18 could run. # JSONB round trip that was crashing the worker before f01f18 could run.
rows = Weather.build_grid_cache_rows(grid_data, valid_time) rows = Weather.build_grid_cache_rows(grid_data, valid_time)
@ -224,6 +247,54 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
end) end)
end end
defp merge_commercial_link_data(grid_data, valid_time) do
# Compute link degradation once per distinct link cluster and cache by
# point. Commercial links only cluster around DFW so most grid points see
# nil — cheap no-op path dominates.
boosted =
Enum.count(grid_data, fn {{lat, lon}, _profile} ->
degradation = Commercial.link_degradation_at({lat, lon}, valid_time)
not is_nil(degradation)
end)
if boosted > 0 do
Logger.info("PropagationGrid: commercial-link degradation available for #{boosted} grid cells")
end
Map.new(grid_data, fn {{lat, lon} = point, profile} ->
case Commercial.link_degradation_at({lat, lon}, valid_time) do
nil -> {point, profile}
degradation -> {point, Map.put(profile, :commercial_link_degradation, degradation)}
end
end)
end
defp merge_nexrad_data(grid_data, valid_time) do
points = Map.keys(grid_data)
case timed("nexrad", fn -> NexradClient.fetch_frame(valid_time, points) end) do
{:ok, observations} ->
apply_nexrad_observations(grid_data, observations)
{:error, reason} ->
Logger.warning("PropagationGrid: NEXRAD fetch failed (continuing without): #{inspect(reason)}")
grid_data
end
end
defp apply_nexrad_observations(grid_data, observations) do
index = Map.new(observations, fn obs -> {{obs.lat, obs.lon}, obs.max_reflectivity_dbz} end)
non_zero = Enum.count(index, fn {_pt, dbz} -> dbz > 0 end)
Logger.info("PropagationGrid: NEXRAD merged (#{non_zero} cells with precip)")
Map.new(grid_data, fn {point, profile} ->
case Map.get(index, point) do
nil -> {point, profile}
dbz -> {point, Map.put(profile, :nexrad_max_reflectivity_dbz, dbz)}
end
end)
end
defp compute_scores(grid_data, valid_time) do defp compute_scores(grid_data, valid_time) do
# Algorithm is the primary scorer. ML score stored in factors for comparison. # Algorithm is the primary scorer. ML score stored in factors for comparison.
compute_scores_algorithm(grid_data, valid_time) compute_scores_algorithm(grid_data, valid_time)

View file

@ -65,6 +65,123 @@ defmodule Microwaveprop.CommercialTest do
end end
end end
describe "link_degradation_at/3" do
@endpoint_dfw %{"lat" => 33.2, "lon" => -96.5}
@far_point {0.0, 0.0}
@near_point {33.2, -96.5}
test "returns nil when no links are within the radius" do
{:ok, _} =
Commercial.create_link(
Map.merge(@link_attrs, %{
endpoint_a: @endpoint_dfw,
endpoint_b: @endpoint_dfw
})
)
valid_time = ~U[2026-04-13 18:00:00Z]
assert Commercial.link_degradation_at(@far_point, valid_time, radius_km: 50) == nil
end
test "returns nil when the link has no recent baseline samples" do
{:ok, _link} =
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
valid_time = ~U[2026-04-13 18:00:00Z]
assert Commercial.link_degradation_at(@near_point, valid_time) == nil
end
test "returns positive degradation when current rx_power is below baseline" do
{:ok, link} =
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
valid_time = ~U[2026-04-13 18:00:00Z]
# 7-day baseline: -52 dBm average
for day_offset <- 1..6 do
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
rx_power_0: -52.0,
rx_power_1: -52.0,
link_state: 1
})
end
# Current (within 15 min) drops to -62 — 10 dB deep fade.
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -300, :second),
rx_power_0: -62.0,
rx_power_1: -62.0,
link_state: 1
})
result = Commercial.link_degradation_at(@near_point, valid_time)
assert result
assert_in_delta result.degradation_db, 10.0, 0.5
assert result.n_links == 1
end
test "returns 0 degradation when rx_power matches baseline" do
{:ok, link} =
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
valid_time = ~U[2026-04-13 18:00:00Z]
for day_offset <- 1..6 do
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
rx_power_0: -55.0,
link_state: 1
})
end
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -60, :second),
rx_power_0: -55.0,
link_state: 1
})
result = Commercial.link_degradation_at(@near_point, valid_time)
assert result
assert_in_delta result.degradation_db, 0.0, 0.5
end
test "ignores links with link_state != 1 (down link isn't tropo)" do
{:ok, link} =
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
valid_time = ~U[2026-04-13 18:00:00Z]
for day_offset <- 1..6 do
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
rx_power_0: -52.0,
link_state: 1
})
end
{:ok, _} =
Commercial.create_sample(%{
link_id: link.id,
sampled_at: DateTime.add(valid_time, -120, :second),
rx_power_0: -90.0,
link_state: 0
})
assert Commercial.link_degradation_at(@near_point, valid_time) == nil
end
end
describe "weather_stations/0" do describe "weather_stations/0" do
test "returns unique weather stations from enabled links" do test "returns unique weather stations from enabled links" do
{:ok, _} = Commercial.create_link(@link_attrs) {:ok, _} = Commercial.create_link(@link_attrs)

View file

@ -86,6 +86,44 @@ defmodule Microwaveprop.Propagation.ScorerTest do
end end
end end
describe "dbz_to_rain_rate_mmhr/1" do
test "returns 0.0 for nil" do
assert Scorer.dbz_to_rain_rate_mmhr(nil) == 0.0
end
test "returns 0.0 for sub-rain reflectivity (below 5 dBZ)" do
# Clear air / ground clutter — not rain.
assert Scorer.dbz_to_rain_rate_mmhr(3.0) == 0.0
assert Scorer.dbz_to_rain_rate_mmhr(0.0) == 0.0
end
test "20 dBZ is light rain ~0.7 mm/hr (Marshall-Palmer)" do
# Z = 10^(dbz/10) = 100; R = (100/200)^(1/1.6) ≈ 0.66 mm/hr
rate = Scorer.dbz_to_rain_rate_mmhr(20.0)
assert_in_delta rate, 0.66, 0.05
end
test "30 dBZ is moderate rain ~2.7 mm/hr" do
# Z = 1000; R = (1000/200)^(1/1.6) ≈ 2.7 mm/hr
rate = Scorer.dbz_to_rain_rate_mmhr(30.0)
assert_in_delta rate, 2.7, 0.2
end
test "45 dBZ is heavy rain ~24 mm/hr" do
# Z = 31623; R = (31623/200)^(1/1.6) ≈ 23.7 mm/hr
rate = Scorer.dbz_to_rain_rate_mmhr(45.0)
assert_in_delta rate, 23.7, 1.0
end
test "55 dBZ clips to a hail-safe ceiling (≤150 mm/hr)" do
# Pure Marshall-Palmer at 55 dBZ returns ~116 mm/hr; at 60 it blows past
# 200 which is unrealistic — hail contamination. Clip to the ITU rain-rate
# tabulated range.
assert Scorer.dbz_to_rain_rate_mmhr(55.0) <= 150.0
assert Scorer.dbz_to_rain_rate_mmhr(65.0) <= 150.0
end
end
# ── score_humidity/2 ────────────────────────────────────────────── # ── score_humidity/2 ──────────────────────────────────────────────
describe "score_humidity/2 beneficial (10 GHz)" do describe "score_humidity/2 beneficial (10 GHz)" do
@ -285,6 +323,80 @@ defmodule Microwaveprop.Propagation.ScorerTest do
end end
end end
describe "score_refractivity/4 with native-profile duct support" do
# New 4-arity variant: adds best_duct_band_ghz from hrrr_native_profiles so
# cells with a thin native duct that supports the target band get a boost
# the HRRR pressure-level gradient alone would miss. Cells where the native
# duct only supports sub-band frequencies do NOT get a boost.
test "native duct supporting the target band boosts the score" do
boosted = Scorer.score_refractivity(-80, 800, 15.0, @band_10g)
plain = Scorer.score_refractivity(-80, 800, nil, @band_10g)
assert boosted > plain
assert boosted <= 100
end
test "native duct below the target band does NOT boost" do
# A 3 GHz duct is useless for 10 GHz.
unchanged = Scorer.score_refractivity(-80, 800, 3.0, @band_10g)
plain = Scorer.score_refractivity(-80, 800, nil, @band_10g)
assert unchanged == plain
end
test "nil best_duct_band_ghz is equivalent to the 3-arity form" do
assert Scorer.score_refractivity(-100, 600, nil, @band_10g) ==
Scorer.score_refractivity(-100, 600, @band_10g)
end
test "nil gradient still returns 50 regardless of duct data" do
assert Scorer.score_refractivity(nil, 600, 15.0, @band_10g) == 50
end
test "24 GHz is boosted only when native duct supports ≥24 GHz" do
weak = Scorer.score_refractivity(-80, 800, 15.0, @band_24g)
strong = Scorer.score_refractivity(-80, 800, 30.0, @band_24g)
plain = Scorer.score_refractivity(-80, 800, nil, @band_24g)
assert weak == plain
assert strong > plain
end
end
describe "commercial_link_boost/2" do
# Inverse sensor: when nearby commercial LOS links are fading below baseline,
# the *same* multipath that hurts them is the enhanced refractivity that
# helps beyond-LOS amateur paths. >8 dB of degradation means multi-Fresnel-
# zone disturbance — strong boost. 3-8 dB is a mild boost. <3 dB is noise.
test "nil degradation is a no-op" do
assert Scorer.commercial_link_boost(75, nil) == 75
end
test "<3 dB fade is noise and does not boost" do
assert Scorer.commercial_link_boost(75, %{degradation_db: 2.0, n_links: 3}) == 75
end
test "5 dB fade applies a mild boost" do
boosted = Scorer.commercial_link_boost(75, %{degradation_db: 5.0, n_links: 3})
assert boosted > 75
assert boosted < 90
end
test "12 dB fade applies a large boost capped at 100" do
boosted = Scorer.commercial_link_boost(75, %{degradation_db: 12.0, n_links: 3})
assert boosted > 85
assert boosted <= 100
end
test "single-link result still boosts (n_links >= 1)" do
boosted = Scorer.commercial_link_boost(70, %{degradation_db: 6.0, n_links: 1})
assert boosted > 70
end
test "empty-links (n_links == 0) is a no-op" do
assert Scorer.commercial_link_boost(75, %{degradation_db: 10.0, n_links: 0}) == 75
end
end
# ── score_sky/1 ────────────────────────────────────────────────── # ── score_sky/1 ──────────────────────────────────────────────────
describe "score_sky/1" do describe "score_sky/1" do

View file

@ -39,6 +39,72 @@ defmodule Microwaveprop.PropagationTest do
assert is_integer(result.band_mhz) assert is_integer(result.band_mhz)
end) end)
end end
test "NEXRAD reflectivity drops the 24 GHz rain score when HRRR precip_mm is 0" do
# Same profile, once without NEXRAD and once with a heavy-rain dBZ. The
# 24 GHz rain-score factor must drop because NEXRAD can report rain that
# HRRR's hourly precip accumulation hasn't caught yet.
base_profile = %{
surface_temp_c: 25.0,
surface_dewpoint_c: 18.0,
surface_pressure_mb: 1013.0,
hpbl_m: 500.0,
wind_u: 3.0,
wind_v: 2.0,
cloud_cover_pct: 15.0,
precip_mm: 0.0,
profile: [
%{"pres" => 1000.0, "tmpc" => 25.0, "dwpc" => 18.0, "hght" => 100.0},
%{"pres" => 950.0, "tmpc" => 19.0, "dwpc" => 10.0, "hght" => 600.0}
]
}
valid_time = ~U[2026-07-15 13:00:00Z]
without = Propagation.score_grid_point(base_profile, valid_time, 33.0, -97.0)
with_nexrad =
base_profile
|> Map.put(:nexrad_max_reflectivity_dbz, 45.0)
|> Propagation.score_grid_point(valid_time, 33.0, -97.0)
rain_dry = Enum.find(without, &(&1.band_mhz == 24_000)).factors.rain
rain_wet = Enum.find(with_nexrad, &(&1.band_mhz == 24_000)).factors.rain
assert rain_wet < rain_dry
end
test "NEXRAD is ignored when HRRR precip_mm already reports heavier rain" do
base_profile = %{
surface_temp_c: 25.0,
surface_dewpoint_c: 18.0,
surface_pressure_mb: 1013.0,
hpbl_m: 500.0,
wind_u: 3.0,
wind_v: 2.0,
cloud_cover_pct: 15.0,
# 30 mm/hr → far above the ~3 mm/hr NEXRAD gives for 30 dBZ, so HRRR wins.
precip_mm: 30.0,
profile: [
%{"pres" => 1000.0, "tmpc" => 25.0, "dwpc" => 18.0, "hght" => 100.0},
%{"pres" => 950.0, "tmpc" => 19.0, "dwpc" => 10.0, "hght" => 600.0}
]
}
valid_time = ~U[2026-07-15 13:00:00Z]
with_light_nexrad =
base_profile
|> Map.put(:nexrad_max_reflectivity_dbz, 30.0)
|> Propagation.score_grid_point(valid_time, 33.0, -97.0)
without = Propagation.score_grid_point(base_profile, valid_time, 33.0, -97.0)
rain_no_nx = Enum.find(without, &(&1.band_mhz == 24_000)).factors.rain
rain_with_nx = Enum.find(with_light_nexrad, &(&1.band_mhz == 24_000)).factors.rain
assert rain_with_nx == rain_no_nx
end
end end
describe "upsert_scores/1" do describe "upsert_scores/1" do

View file

@ -29,6 +29,21 @@ defmodule Microwaveprop.Weather.Era5BatchClientTest do
end end
describe "fetch_month_into_db/1 idempotency" do describe "fetch_month_into_db/1 idempotency" do
# This describe block exercises the uncached path which would otherwise
# hit the real CDS API if the developer has ERA5_CDS_API_KEY set via
# direnv. Clear the env var for these tests so the expected "API key not
# configured" short-circuit fires and the tests stay hermetic.
setup do
previous = System.get_env("ERA5_CDS_API_KEY")
System.delete_env("ERA5_CDS_API_KEY")
on_exit(fn ->
if previous, do: System.put_env("ERA5_CDS_API_KEY", previous)
end)
:ok
end
test "returns {:ok, 0} when the month-tile already has any profile" do test "returns {:ok, 0} when the month-tile already has any profile" do
# Seed one profile inside the 2014-03, tile (32, -98) window. # Seed one profile inside the 2014-03, tile (32, -98) window.
%Era5Profile{} %Era5Profile{}