Wire NEXRAD, native ducts, and commercial links into scoring
Three signal sources we already collect but weren't using: * NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken as max of HRRR-derived and NEXRAD-derived rate so fast convective cells between HRRR hourly analyses can still trigger the rain penalty. Only active on f00 — forecast hours can't see future radar. New Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr hail-safe ceiling. * hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4 applies a 1.15× boost when the cell's native-resolution duct supports the target band's frequency. HRRR pressure-level gradients systematically under-read thin trapping layers the native profile can resolve. Sub-band ducts do NOT boost — they're evidence that the gradient we have is all there is at the target frequency. * Commercial LOS link rx_power fading → inverse tropo sensor. Commercial.link_degradation_at/3 computes the average 7-day-baseline vs current delta across enabled links within 75 km, ignoring links where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25 to the composite score for 3+ dB of fading. ~150 km radius around DFW is the only zone this helps today, but it's the first *measured* signal in the algorithm vs the model-derived proxies. Also fix a latent test bug exposed by the earlier ERA5 poll-timeout bump: era5_batch_client_test's "uncached path returns error" tests hung for up to an hour when run with direnv's real CDS key. New describe-level setup explicitly unsets the env var so the tests stay hermetic. 1,359 tests, 0 failures.
This commit is contained in:
parent
bc6377feb3
commit
86364f43aa
9 changed files with 693 additions and 8 deletions
38
algo.md
38
algo.md
|
|
@ -766,12 +766,46 @@ Per the user's directive to model **all bands ≥902 MHz**, these are added to `
|
|||
### Open items / next analyses
|
||||
|
||||
1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny.
|
||||
2. **NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring.
|
||||
3. **`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude.
|
||||
2. ~~**NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring.~~ **Landed.** See "NEXRAD composite reflectivity" below.
|
||||
3. ~~**`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude.~~ **Landed.** See "Native-profile duct boost" below.
|
||||
4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band.
|
||||
|
||||
A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL.
|
||||
|
||||
### NEXRAD composite reflectivity → rain-attenuation score
|
||||
|
||||
HRRR hourly precipitation accumulation lags fast-moving convective cells by up to 59 minutes. NEXRAD n0q composite reflectivity runs at 5-minute cadence and catches those cells at the moment they pass over a grid cell. The scoring pipeline now takes the *max* of the HRRR-derived rain rate (from `precip_mm`) and the NEXRAD-derived rain rate (from `max_reflectivity_dbz` via Marshall-Palmer) so either source can trigger the rain penalty without double-counting.
|
||||
|
||||
`Scorer.dbz_to_rain_rate_mmhr/1` implements:
|
||||
|
||||
```
|
||||
R (mm/hr) = (Z / 200)^(1/1.6) where Z = 10^(dBZ/10)
|
||||
```
|
||||
|
||||
with a 5 dBZ noise floor (ground clutter / clear air) and a 150 mm/hr ceiling (hail contamination). Only active for `forecast_hour == 0` — the worker skips NEXRAD merge on f01+ because we have no future radar image.
|
||||
|
||||
### Native-profile duct boost
|
||||
|
||||
The base refractivity score uses HRRR's 13 pressure levels, which systematically under-read thin trapping layers (see Part 2c — only 0.12% of native-resolution cells support ≥15 GHz, but the pressure-level data looks the same above and below that threshold). When `hrrr_native_profiles.best_duct_band_ghz` is present and its value is ≥ the target band's frequency, `Scorer.score_refractivity/4` multiplies the base score by 1.15×. A duct that only supports sub-band frequencies does *not* boost — it's evidence that the gradient we have is all there is at the target band.
|
||||
|
||||
This is an additive correction to the HPBL multiplier introduced earlier in Part 2c. The two multipliers compose: a thin shallow-BL cell with a 24-GHz-supporting native duct gets the full 1.05–1.10× HPBL bonus *and* the 1.15× native-duct bonus, clamped at 100.
|
||||
|
||||
### Commercial-link inverse sensor
|
||||
|
||||
Seven af11x / af60 commercial microwave links around Princeton TX (33.2°N, 96.5°W) are polled every 5 minutes via SNMP and stored in `commercial_samples`. These are short (2–6 km) LOS paths where the same refractivity anomaly that helps beyond-LOS amateur propagation degrades the link's rx_power through multipath fading.
|
||||
|
||||
`Commercial.link_degradation_at/3` computes the average of (7-day baseline rx_power − current rx_power) across all healthy (`link_state == 1`) links within a configurable radius of a target point (default 75 km). The worker calls this for every grid cell on `forecast_hour == 0` runs; commercial links only exist near DFW so almost every cell gets `nil` at near-zero cost.
|
||||
|
||||
`Scorer.commercial_link_boost/2` then adds a terminal boost to the composite score:
|
||||
|
||||
| degradation_db | Bonus | Interpretation |
|
||||
|---|---|---|
|
||||
| <3 dB | 0 | Noise floor |
|
||||
| 3–8 dB | +2 to +10 | Mild multipath — weak enhancement |
|
||||
| ≥8 dB | +10 to +25 (clamped ≤100) | Deep fade — strong ducting signature |
|
||||
|
||||
This is the first *measured* signal in the algorithm — every other factor is a model-derived proxy. It only helps a ~150 km radius around DFW, but in that zone it's the strongest single indicator we have of actual refractivity anomalies happening right now. Out-of-zone cells see no change.
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Band Configuration
|
||||
|
|
|
|||
|
|
@ -7,6 +7,11 @@ defmodule Microwaveprop.Commercial do
|
|||
alias Microwaveprop.Commercial.Sample
|
||||
alias Microwaveprop.Repo
|
||||
|
||||
@default_radius_km 75.0
|
||||
@default_baseline_days 7
|
||||
@default_current_window_seconds 900
|
||||
@min_baseline_samples 5
|
||||
|
||||
@spec enabled_links() :: [Link.t()]
|
||||
def enabled_links do
|
||||
Link
|
||||
|
|
@ -14,6 +19,142 @@ defmodule Microwaveprop.Commercial do
|
|||
|> Repo.all()
|
||||
end
|
||||
|
||||
@doc """
|
||||
Returns the aggregate commercial-link rx_power degradation at a location,
|
||||
or `nil` if no healthy links are in range.
|
||||
|
||||
Commercial LOS microwave links near the target point act as an inverse
|
||||
tropo sensor: when their rx_power drops significantly below the 7-day
|
||||
baseline without an obvious equipment cause (link_state still 1), the
|
||||
refractivity structure that hurts their short Fresnel zone is usually
|
||||
the same structure that *helps* long beyond-LOS amateur propagation.
|
||||
|
||||
Result map:
|
||||
%{
|
||||
degradation_db: float (positive means worse than baseline),
|
||||
baseline_dbm: float,
|
||||
current_dbm: float,
|
||||
n_links: integer
|
||||
}
|
||||
|
||||
## Options
|
||||
|
||||
* `:radius_km` — distance from the target point to include a link (default 75)
|
||||
* `:baseline_days` — days of history for the baseline average (default 7)
|
||||
* `:current_window_seconds` — how recent "current" rx must be (default 900)
|
||||
"""
|
||||
@spec link_degradation_at({float(), float()}, DateTime.t(), keyword()) :: map() | nil
|
||||
def link_degradation_at({lat, lon}, valid_time, opts \\ []) do
|
||||
radius_km = Keyword.get(opts, :radius_km, @default_radius_km)
|
||||
baseline_days = Keyword.get(opts, :baseline_days, @default_baseline_days)
|
||||
current_window = Keyword.get(opts, :current_window_seconds, @default_current_window_seconds)
|
||||
|
||||
candidates =
|
||||
enabled_links()
|
||||
|> Enum.map(fn link -> {link, link_endpoint(link)} end)
|
||||
|> Enum.reject(fn {_link, endpoint} -> is_nil(endpoint) end)
|
||||
|> Enum.filter(fn {_link, {elat, elon}} ->
|
||||
haversine_km(lat, lon, elat, elon) <= radius_km
|
||||
end)
|
||||
|
||||
baseline_cutoff = DateTime.add(valid_time, -baseline_days * 24 * 3600, :second)
|
||||
current_cutoff = DateTime.add(valid_time, -current_window, :second)
|
||||
|
||||
results =
|
||||
candidates
|
||||
|> Enum.map(fn {link, _endpoint} ->
|
||||
link_degradation(link.id, baseline_cutoff, current_cutoff, valid_time)
|
||||
end)
|
||||
|> Enum.reject(&is_nil/1)
|
||||
|
||||
case results do
|
||||
[] ->
|
||||
nil
|
||||
|
||||
list ->
|
||||
baseline_avg = average(Enum.map(list, & &1.baseline_dbm))
|
||||
current_avg = average(Enum.map(list, & &1.current_dbm))
|
||||
|
||||
%{
|
||||
degradation_db: Float.round(baseline_avg - current_avg, 2),
|
||||
baseline_dbm: Float.round(baseline_avg, 2),
|
||||
current_dbm: Float.round(current_avg, 2),
|
||||
n_links: length(list)
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
defp link_degradation(link_id, baseline_cutoff, current_cutoff, valid_time) do
|
||||
baseline =
|
||||
Sample
|
||||
|> where([s], s.link_id == ^link_id)
|
||||
|> where([s], s.sampled_at >= ^baseline_cutoff and s.sampled_at < ^current_cutoff)
|
||||
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|
||||
|> select([s], avg(s.rx_power_0))
|
||||
|> Repo.one()
|
||||
|
||||
{current, baseline_n} = fetch_current_and_baseline_count(link_id, current_cutoff, valid_time)
|
||||
|
||||
cond do
|
||||
is_nil(baseline) or is_nil(current) ->
|
||||
nil
|
||||
|
||||
baseline_n < @min_baseline_samples ->
|
||||
nil
|
||||
|
||||
true ->
|
||||
%{baseline_dbm: to_float(baseline), current_dbm: current}
|
||||
end
|
||||
end
|
||||
|
||||
defp to_float(%Decimal{} = d), do: Decimal.to_float(d)
|
||||
defp to_float(n) when is_number(n), do: n * 1.0
|
||||
|
||||
defp fetch_current_and_baseline_count(link_id, current_cutoff, valid_time) do
|
||||
current =
|
||||
Sample
|
||||
|> where([s], s.link_id == ^link_id)
|
||||
|> where([s], s.sampled_at >= ^current_cutoff and s.sampled_at <= ^valid_time)
|
||||
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|
||||
|> order_by([s], desc: s.sampled_at)
|
||||
|> limit(1)
|
||||
|> select([s], s.rx_power_0)
|
||||
|> Repo.one()
|
||||
|
||||
baseline_n =
|
||||
Sample
|
||||
|> where([s], s.link_id == ^link_id)
|
||||
|> where([s], s.sampled_at < ^current_cutoff)
|
||||
|> where([s], s.link_state == 1 and not is_nil(s.rx_power_0))
|
||||
|> select([s], count(s.id))
|
||||
|> Repo.one()
|
||||
|
||||
{current, baseline_n}
|
||||
end
|
||||
|
||||
defp link_endpoint(%Link{endpoint_a: %{"lat" => lat, "lon" => lon}}) when is_number(lat) and is_number(lon),
|
||||
do: {lat * 1.0, lon * 1.0}
|
||||
|
||||
defp link_endpoint(_), do: nil
|
||||
|
||||
defp haversine_km(lat1, lon1, lat2, lon2) do
|
||||
r = 6371.0
|
||||
phi1 = lat1 * :math.pi() / 180
|
||||
phi2 = lat2 * :math.pi() / 180
|
||||
dphi = (lat2 - lat1) * :math.pi() / 180
|
||||
dlambda = (lon2 - lon1) * :math.pi() / 180
|
||||
|
||||
a =
|
||||
:math.sin(dphi / 2) * :math.sin(dphi / 2) +
|
||||
:math.cos(phi1) * :math.cos(phi2) * :math.sin(dlambda / 2) * :math.sin(dlambda / 2)
|
||||
|
||||
c = 2 * :math.atan2(:math.sqrt(a), :math.sqrt(1 - a))
|
||||
r * c
|
||||
end
|
||||
|
||||
defp average([]), do: 0.0
|
||||
defp average(list), do: Enum.sum(list) / length(list)
|
||||
|
||||
@spec create_link(map()) :: {:ok, Link.t()} | {:error, Ecto.Changeset.t()}
|
||||
def create_link(attrs) do
|
||||
%Link{}
|
||||
|
|
|
|||
|
|
@ -90,10 +90,11 @@ defmodule Microwaveprop.Propagation do
|
|||
longitude: longitude,
|
||||
pressure_mb: hrrr_profile.surface_pressure_mb,
|
||||
prev_pressure_mb: nil,
|
||||
rain_rate_mmhr: Scorer.precip_to_rate_mmhr(hrrr_profile[:precip_mm]),
|
||||
rain_rate_mmhr: merged_rain_rate(hrrr_profile),
|
||||
min_refractivity_gradient: hrrr_profile[:native_min_gradient] || derived[:min_refractivity_gradient],
|
||||
bl_depth_m: hrrr_profile[:hpbl_m],
|
||||
pwat_mm: hrrr_profile[:pwat_mm]
|
||||
pwat_mm: hrrr_profile[:pwat_mm],
|
||||
best_duct_band_ghz: hrrr_profile[:best_duct_freq_ghz] || hrrr_profile[:best_duct_band_ghz]
|
||||
}
|
||||
|
||||
duct_info =
|
||||
|
|
@ -106,13 +107,45 @@ defmodule Microwaveprop.Propagation do
|
|||
}
|
||||
end
|
||||
|
||||
link_degradation = hrrr_profile[:commercial_link_degradation]
|
||||
|
||||
Enum.map(BandConfig.all_bands(), fn band_config ->
|
||||
result = Scorer.composite_score(conditions, band_config)
|
||||
result = Map.put(result, :band_mhz, band_config.freq_mhz)
|
||||
|
||||
boosted_score =
|
||||
if link_degradation do
|
||||
Scorer.commercial_link_boost(result.score, link_degradation)
|
||||
else
|
||||
result.score
|
||||
end
|
||||
|
||||
result =
|
||||
result
|
||||
|> Map.put(:score, boosted_score)
|
||||
|> Map.put(:band_mhz, band_config.freq_mhz)
|
||||
|
||||
result =
|
||||
if link_degradation do
|
||||
put_in(result, [:factors, :commercial_link_degradation], link_degradation)
|
||||
else
|
||||
result
|
||||
end
|
||||
|
||||
if duct_info, do: put_in(result, [:factors, :duct_info], duct_info), else: result
|
||||
end)
|
||||
end
|
||||
|
||||
# Pick the heavier of HRRR's hourly accumulation-derived rate and NEXRAD's
|
||||
# reflectivity-derived rate. NEXRAD catches fast-moving convective cells that
|
||||
# fall between HRRR hourly analyses; HRRR catches broad stratiform rain that
|
||||
# NEXRAD reports as low dBZ. Taking max lets either source trigger the rain
|
||||
# penalty without double-counting.
|
||||
defp merged_rain_rate(hrrr_profile) do
|
||||
hrrr_rate = Scorer.precip_to_rate_mmhr(hrrr_profile[:precip_mm])
|
||||
nexrad_rate = Scorer.dbz_to_rain_rate_mmhr(hrrr_profile[:nexrad_max_reflectivity_dbz])
|
||||
max(hrrr_rate, nexrad_rate)
|
||||
end
|
||||
|
||||
@doc """
|
||||
Upsert propagation scores in batches within a transaction so readers see all-or-nothing.
|
||||
|
||||
|
|
|
|||
|
|
@ -50,6 +50,27 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
def precip_to_rate_mmhr(mm) when mm > 0, do: mm / 1
|
||||
def precip_to_rate_mmhr(_mm), do: 0.0
|
||||
|
||||
@doc """
|
||||
Converts NEXRAD composite reflectivity (dBZ) to rain rate (mm/hr) via the
|
||||
Marshall-Palmer Z-R relationship `Z = 200 * R^1.6`, so `R = (Z/200)^(1/1.6)`
|
||||
with `Z = 10^(dBZ/10)`.
|
||||
|
||||
Clipping rules:
|
||||
* Below 5 dBZ — not rain (ground clutter / clear air). Returns 0.0.
|
||||
* Above 150 mm/hr — hail contamination (55+ dBZ can return >100 mm/hr
|
||||
from pure Marshall-Palmer). The ITU-R P.838 rain table tops out at
|
||||
~150 mm/hr so we clip there to stay inside the calibrated regime.
|
||||
"""
|
||||
@spec dbz_to_rain_rate_mmhr(number() | nil) :: float()
|
||||
def dbz_to_rain_rate_mmhr(nil), do: 0.0
|
||||
def dbz_to_rain_rate_mmhr(dbz) when dbz < 5.0, do: 0.0
|
||||
|
||||
def dbz_to_rain_rate_mmhr(dbz) do
|
||||
z = :math.pow(10, dbz / 10)
|
||||
r = :math.pow(z / 200, 1 / 1.6)
|
||||
min(r, 150.0)
|
||||
end
|
||||
|
||||
# ── Factor 1: Humidity ────────────────────────────────────────────
|
||||
|
||||
@doc """
|
||||
|
|
@ -163,9 +184,27 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
applied to *both* the threshold-matched score and the default fallback.
|
||||
"""
|
||||
@spec score_refractivity(number() | nil, number() | nil, map()) :: integer()
|
||||
def score_refractivity(nil, _bl_depth_m, _band_config), do: 50
|
||||
def score_refractivity(min_gradient, bl_depth_m, band_config) do
|
||||
score_refractivity(min_gradient, bl_depth_m, nil, band_config)
|
||||
end
|
||||
|
||||
def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do
|
||||
@doc """
|
||||
Scores refractivity with optional native-profile duct info.
|
||||
|
||||
`best_duct_band_ghz` comes from `hrrr_native_profiles` and represents the
|
||||
highest frequency the cell's native-resolution duct can trap. When it's
|
||||
≥ the target band's frequency the base score is boosted 1.15× because
|
||||
HRRR pressure-level gradients systematically under-read thin ducts the
|
||||
native profile can resolve (see Part 2c Apr 13 2026 findings).
|
||||
|
||||
A duct that only supports lower frequencies does NOT boost the score —
|
||||
it's a signal that the gradient we have is *all there is* at the target
|
||||
band.
|
||||
"""
|
||||
@spec score_refractivity(number() | nil, number() | nil, number() | nil, map()) :: integer()
|
||||
def score_refractivity(nil, _bl_depth_m, _best_duct_band_ghz, _band_config), do: 50
|
||||
|
||||
def score_refractivity(min_gradient, bl_depth_m, best_duct_band_ghz, %{freq_mhz: freq_mhz, humidity_effect: effect}) do
|
||||
thresholds = BandConfig.refractivity_thresholds()
|
||||
|
||||
base =
|
||||
|
|
@ -178,7 +217,9 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
if effect == :beneficial, do: beneficial_default, else: harmful_default
|
||||
end
|
||||
|
||||
apply_hpbl_multiplier(base, bl_depth_m)
|
||||
base
|
||||
|> apply_hpbl_multiplier(bl_depth_m)
|
||||
|> apply_native_duct_boost(best_duct_band_ghz, freq_mhz)
|
||||
end
|
||||
|
||||
# HPBL multiplier — applied to the base refractivity score. Calibrated to the
|
||||
|
|
@ -192,6 +233,60 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92)
|
||||
defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78)
|
||||
|
||||
# Native-profile duct boost — 1.15× when the cell's best duct supports the
|
||||
# target band's frequency. HRRR pressure-level gradients systematically
|
||||
# under-read thin ducts (~250m vertical spacing vs 50 native hybrid levels),
|
||||
# so when hrrr_native_profiles reports a duct band at or above the target
|
||||
# frequency, the base gradient score is under-estimating the real channel.
|
||||
defp apply_native_duct_boost(base, nil, _freq_mhz), do: base
|
||||
|
||||
defp apply_native_duct_boost(base, best_duct_band_ghz, freq_mhz) do
|
||||
target_ghz = freq_mhz / 1_000
|
||||
|
||||
if best_duct_band_ghz >= target_ghz do
|
||||
clamp_score(base * 1.15)
|
||||
else
|
||||
base
|
||||
end
|
||||
end
|
||||
|
||||
@doc """
|
||||
Inverse-sensor boost from commercial LOS link degradation.
|
||||
|
||||
Nearby short-path commercial links (11/24/68 GHz in the DFW area) act as a
|
||||
physical refractivity anomaly sensor: when their rx_power drops 5+ dB below
|
||||
the 7-day baseline *without* an equipment cause, the same multipath fading
|
||||
that hurts their short Fresnel zone is what carries the beyond-LOS paths.
|
||||
This is a terminal boost applied once per composite score, not per-band —
|
||||
the underlying physics is band-agnostic at the resolution we can measure.
|
||||
|
||||
Takes the incoming composite (or any 0-100) score and a degradation map
|
||||
from `Commercial.link_degradation_at/3`; returns a boosted integer score.
|
||||
|
||||
* nil or empty result (`n_links == 0`) — no-op
|
||||
* <3 dB — noise floor, no-op
|
||||
* 3–8 dB — mild boost (+0 to +10)
|
||||
* ≥8 dB — strong boost (+10 to +25, clamped ≤ 100)
|
||||
"""
|
||||
@spec commercial_link_boost(number(), map() | nil) :: integer()
|
||||
def commercial_link_boost(score, nil), do: clamp_score(score)
|
||||
|
||||
def commercial_link_boost(score, %{n_links: 0}), do: clamp_score(score)
|
||||
|
||||
def commercial_link_boost(score, %{degradation_db: db}) when db < 3.0, do: clamp_score(score)
|
||||
|
||||
def commercial_link_boost(score, %{degradation_db: db}) when db < 8.0 do
|
||||
# 3 dB → +2; 7.9 dB → ~+10 (linear interpolation)
|
||||
bonus = 2 + (db - 3.0) * 8.0 / 5.0
|
||||
clamp_score(score + bonus)
|
||||
end
|
||||
|
||||
def commercial_link_boost(score, %{degradation_db: db}) do
|
||||
# 8 dB → +10; 16 dB → +25 (linear)
|
||||
bonus = min(25.0, 10 + (db - 8.0) * 15.0 / 8.0)
|
||||
clamp_score(score + bonus)
|
||||
end
|
||||
|
||||
defp clamp_score(value) do
|
||||
value |> round() |> max(0) |> min(100)
|
||||
end
|
||||
|
|
@ -381,6 +476,7 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
score_refractivity(
|
||||
conditions.min_refractivity_gradient,
|
||||
conditions.bl_depth_m,
|
||||
conditions[:best_duct_band_ghz],
|
||||
band_config
|
||||
),
|
||||
sky: score_sky(conditions.sky_cover_pct),
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
max_attempts: 3,
|
||||
unique: [period: 10_800, states: [:available, :scheduled, :executing, :retryable]]
|
||||
|
||||
alias Microwaveprop.Commercial
|
||||
alias Microwaveprop.Propagation
|
||||
alias Microwaveprop.Propagation.BandConfig
|
||||
alias Microwaveprop.Propagation.Grid
|
||||
|
|
@ -20,6 +21,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
alias Microwaveprop.Weather.GridCache
|
||||
alias Microwaveprop.Weather.HrrrClient
|
||||
alias Microwaveprop.Weather.HrrrNativeClient
|
||||
alias Microwaveprop.Weather.NexradClient
|
||||
alias Microwaveprop.Weather.SoundingParams
|
||||
|
||||
require Logger
|
||||
|
|
@ -93,6 +95,27 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
grid_data = merge_native_duct_data(grid_data, run_time, forecast_hour)
|
||||
:erlang.garbage_collect()
|
||||
|
||||
# NEXRAD current-hour composite reflectivity catches fast-moving
|
||||
# convective cells between HRRR hourly analyses. Only useful for
|
||||
# f00 — forecast hours can't see the future radar image.
|
||||
grid_data =
|
||||
if forecast_hour == 0 do
|
||||
merge_nexrad_data(grid_data, valid_time)
|
||||
else
|
||||
grid_data
|
||||
end
|
||||
|
||||
# Commercial-link inverse sensor — only meaningful for f00 because
|
||||
# the measurement is of the current atmospheric state, not a forecast.
|
||||
grid_data =
|
||||
if forecast_hour == 0 do
|
||||
merge_commercial_link_data(grid_data, valid_time)
|
||||
else
|
||||
grid_data
|
||||
end
|
||||
|
||||
:erlang.garbage_collect()
|
||||
|
||||
# Build weather cache rows from in-memory grid_data — avoids the ~20s
|
||||
# JSONB round trip that was crashing the worker before f01–f18 could run.
|
||||
rows = Weather.build_grid_cache_rows(grid_data, valid_time)
|
||||
|
|
@ -224,6 +247,54 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
end)
|
||||
end
|
||||
|
||||
defp merge_commercial_link_data(grid_data, valid_time) do
|
||||
# Compute link degradation once per distinct link cluster and cache by
|
||||
# point. Commercial links only cluster around DFW so most grid points see
|
||||
# nil — cheap no-op path dominates.
|
||||
boosted =
|
||||
Enum.count(grid_data, fn {{lat, lon}, _profile} ->
|
||||
degradation = Commercial.link_degradation_at({lat, lon}, valid_time)
|
||||
not is_nil(degradation)
|
||||
end)
|
||||
|
||||
if boosted > 0 do
|
||||
Logger.info("PropagationGrid: commercial-link degradation available for #{boosted} grid cells")
|
||||
end
|
||||
|
||||
Map.new(grid_data, fn {{lat, lon} = point, profile} ->
|
||||
case Commercial.link_degradation_at({lat, lon}, valid_time) do
|
||||
nil -> {point, profile}
|
||||
degradation -> {point, Map.put(profile, :commercial_link_degradation, degradation)}
|
||||
end
|
||||
end)
|
||||
end
|
||||
|
||||
defp merge_nexrad_data(grid_data, valid_time) do
|
||||
points = Map.keys(grid_data)
|
||||
|
||||
case timed("nexrad", fn -> NexradClient.fetch_frame(valid_time, points) end) do
|
||||
{:ok, observations} ->
|
||||
apply_nexrad_observations(grid_data, observations)
|
||||
|
||||
{:error, reason} ->
|
||||
Logger.warning("PropagationGrid: NEXRAD fetch failed (continuing without): #{inspect(reason)}")
|
||||
grid_data
|
||||
end
|
||||
end
|
||||
|
||||
defp apply_nexrad_observations(grid_data, observations) do
|
||||
index = Map.new(observations, fn obs -> {{obs.lat, obs.lon}, obs.max_reflectivity_dbz} end)
|
||||
non_zero = Enum.count(index, fn {_pt, dbz} -> dbz > 0 end)
|
||||
Logger.info("PropagationGrid: NEXRAD merged (#{non_zero} cells with precip)")
|
||||
|
||||
Map.new(grid_data, fn {point, profile} ->
|
||||
case Map.get(index, point) do
|
||||
nil -> {point, profile}
|
||||
dbz -> {point, Map.put(profile, :nexrad_max_reflectivity_dbz, dbz)}
|
||||
end
|
||||
end)
|
||||
end
|
||||
|
||||
defp compute_scores(grid_data, valid_time) do
|
||||
# Algorithm is the primary scorer. ML score stored in factors for comparison.
|
||||
compute_scores_algorithm(grid_data, valid_time)
|
||||
|
|
|
|||
|
|
@ -65,6 +65,123 @@ defmodule Microwaveprop.CommercialTest do
|
|||
end
|
||||
end
|
||||
|
||||
describe "link_degradation_at/3" do
|
||||
@endpoint_dfw %{"lat" => 33.2, "lon" => -96.5}
|
||||
@far_point {0.0, 0.0}
|
||||
@near_point {33.2, -96.5}
|
||||
|
||||
test "returns nil when no links are within the radius" do
|
||||
{:ok, _} =
|
||||
Commercial.create_link(
|
||||
Map.merge(@link_attrs, %{
|
||||
endpoint_a: @endpoint_dfw,
|
||||
endpoint_b: @endpoint_dfw
|
||||
})
|
||||
)
|
||||
|
||||
valid_time = ~U[2026-04-13 18:00:00Z]
|
||||
assert Commercial.link_degradation_at(@far_point, valid_time, radius_km: 50) == nil
|
||||
end
|
||||
|
||||
test "returns nil when the link has no recent baseline samples" do
|
||||
{:ok, _link} =
|
||||
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
|
||||
|
||||
valid_time = ~U[2026-04-13 18:00:00Z]
|
||||
assert Commercial.link_degradation_at(@near_point, valid_time) == nil
|
||||
end
|
||||
|
||||
test "returns positive degradation when current rx_power is below baseline" do
|
||||
{:ok, link} =
|
||||
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
|
||||
|
||||
valid_time = ~U[2026-04-13 18:00:00Z]
|
||||
|
||||
# 7-day baseline: -52 dBm average
|
||||
for day_offset <- 1..6 do
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
|
||||
rx_power_0: -52.0,
|
||||
rx_power_1: -52.0,
|
||||
link_state: 1
|
||||
})
|
||||
end
|
||||
|
||||
# Current (within 15 min) drops to -62 — 10 dB deep fade.
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -300, :second),
|
||||
rx_power_0: -62.0,
|
||||
rx_power_1: -62.0,
|
||||
link_state: 1
|
||||
})
|
||||
|
||||
result = Commercial.link_degradation_at(@near_point, valid_time)
|
||||
assert result
|
||||
assert_in_delta result.degradation_db, 10.0, 0.5
|
||||
assert result.n_links == 1
|
||||
end
|
||||
|
||||
test "returns 0 degradation when rx_power matches baseline" do
|
||||
{:ok, link} =
|
||||
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
|
||||
|
||||
valid_time = ~U[2026-04-13 18:00:00Z]
|
||||
|
||||
for day_offset <- 1..6 do
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
|
||||
rx_power_0: -55.0,
|
||||
link_state: 1
|
||||
})
|
||||
end
|
||||
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -60, :second),
|
||||
rx_power_0: -55.0,
|
||||
link_state: 1
|
||||
})
|
||||
|
||||
result = Commercial.link_degradation_at(@near_point, valid_time)
|
||||
assert result
|
||||
assert_in_delta result.degradation_db, 0.0, 0.5
|
||||
end
|
||||
|
||||
test "ignores links with link_state != 1 (down link isn't tropo)" do
|
||||
{:ok, link} =
|
||||
Commercial.create_link(Map.merge(@link_attrs, %{endpoint_a: @endpoint_dfw, endpoint_b: @endpoint_dfw}))
|
||||
|
||||
valid_time = ~U[2026-04-13 18:00:00Z]
|
||||
|
||||
for day_offset <- 1..6 do
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -day_offset * 3600 * 24, :second),
|
||||
rx_power_0: -52.0,
|
||||
link_state: 1
|
||||
})
|
||||
end
|
||||
|
||||
{:ok, _} =
|
||||
Commercial.create_sample(%{
|
||||
link_id: link.id,
|
||||
sampled_at: DateTime.add(valid_time, -120, :second),
|
||||
rx_power_0: -90.0,
|
||||
link_state: 0
|
||||
})
|
||||
|
||||
assert Commercial.link_degradation_at(@near_point, valid_time) == nil
|
||||
end
|
||||
end
|
||||
|
||||
describe "weather_stations/0" do
|
||||
test "returns unique weather stations from enabled links" do
|
||||
{:ok, _} = Commercial.create_link(@link_attrs)
|
||||
|
|
|
|||
|
|
@ -86,6 +86,44 @@ defmodule Microwaveprop.Propagation.ScorerTest do
|
|||
end
|
||||
end
|
||||
|
||||
describe "dbz_to_rain_rate_mmhr/1" do
|
||||
test "returns 0.0 for nil" do
|
||||
assert Scorer.dbz_to_rain_rate_mmhr(nil) == 0.0
|
||||
end
|
||||
|
||||
test "returns 0.0 for sub-rain reflectivity (below 5 dBZ)" do
|
||||
# Clear air / ground clutter — not rain.
|
||||
assert Scorer.dbz_to_rain_rate_mmhr(3.0) == 0.0
|
||||
assert Scorer.dbz_to_rain_rate_mmhr(0.0) == 0.0
|
||||
end
|
||||
|
||||
test "20 dBZ is light rain ~0.7 mm/hr (Marshall-Palmer)" do
|
||||
# Z = 10^(dbz/10) = 100; R = (100/200)^(1/1.6) ≈ 0.66 mm/hr
|
||||
rate = Scorer.dbz_to_rain_rate_mmhr(20.0)
|
||||
assert_in_delta rate, 0.66, 0.05
|
||||
end
|
||||
|
||||
test "30 dBZ is moderate rain ~2.7 mm/hr" do
|
||||
# Z = 1000; R = (1000/200)^(1/1.6) ≈ 2.7 mm/hr
|
||||
rate = Scorer.dbz_to_rain_rate_mmhr(30.0)
|
||||
assert_in_delta rate, 2.7, 0.2
|
||||
end
|
||||
|
||||
test "45 dBZ is heavy rain ~24 mm/hr" do
|
||||
# Z = 31623; R = (31623/200)^(1/1.6) ≈ 23.7 mm/hr
|
||||
rate = Scorer.dbz_to_rain_rate_mmhr(45.0)
|
||||
assert_in_delta rate, 23.7, 1.0
|
||||
end
|
||||
|
||||
test "55 dBZ clips to a hail-safe ceiling (≤150 mm/hr)" do
|
||||
# Pure Marshall-Palmer at 55 dBZ returns ~116 mm/hr; at 60 it blows past
|
||||
# 200 which is unrealistic — hail contamination. Clip to the ITU rain-rate
|
||||
# tabulated range.
|
||||
assert Scorer.dbz_to_rain_rate_mmhr(55.0) <= 150.0
|
||||
assert Scorer.dbz_to_rain_rate_mmhr(65.0) <= 150.0
|
||||
end
|
||||
end
|
||||
|
||||
# ── score_humidity/2 ──────────────────────────────────────────────
|
||||
|
||||
describe "score_humidity/2 beneficial (10 GHz)" do
|
||||
|
|
@ -285,6 +323,80 @@ defmodule Microwaveprop.Propagation.ScorerTest do
|
|||
end
|
||||
end
|
||||
|
||||
describe "score_refractivity/4 with native-profile duct support" do
|
||||
# New 4-arity variant: adds best_duct_band_ghz from hrrr_native_profiles so
|
||||
# cells with a thin native duct that supports the target band get a boost
|
||||
# the HRRR pressure-level gradient alone would miss. Cells where the native
|
||||
# duct only supports sub-band frequencies do NOT get a boost.
|
||||
|
||||
test "native duct supporting the target band boosts the score" do
|
||||
boosted = Scorer.score_refractivity(-80, 800, 15.0, @band_10g)
|
||||
plain = Scorer.score_refractivity(-80, 800, nil, @band_10g)
|
||||
assert boosted > plain
|
||||
assert boosted <= 100
|
||||
end
|
||||
|
||||
test "native duct below the target band does NOT boost" do
|
||||
# A 3 GHz duct is useless for 10 GHz.
|
||||
unchanged = Scorer.score_refractivity(-80, 800, 3.0, @band_10g)
|
||||
plain = Scorer.score_refractivity(-80, 800, nil, @band_10g)
|
||||
assert unchanged == plain
|
||||
end
|
||||
|
||||
test "nil best_duct_band_ghz is equivalent to the 3-arity form" do
|
||||
assert Scorer.score_refractivity(-100, 600, nil, @band_10g) ==
|
||||
Scorer.score_refractivity(-100, 600, @band_10g)
|
||||
end
|
||||
|
||||
test "nil gradient still returns 50 regardless of duct data" do
|
||||
assert Scorer.score_refractivity(nil, 600, 15.0, @band_10g) == 50
|
||||
end
|
||||
|
||||
test "24 GHz is boosted only when native duct supports ≥24 GHz" do
|
||||
weak = Scorer.score_refractivity(-80, 800, 15.0, @band_24g)
|
||||
strong = Scorer.score_refractivity(-80, 800, 30.0, @band_24g)
|
||||
plain = Scorer.score_refractivity(-80, 800, nil, @band_24g)
|
||||
assert weak == plain
|
||||
assert strong > plain
|
||||
end
|
||||
end
|
||||
|
||||
describe "commercial_link_boost/2" do
|
||||
# Inverse sensor: when nearby commercial LOS links are fading below baseline,
|
||||
# the *same* multipath that hurts them is the enhanced refractivity that
|
||||
# helps beyond-LOS amateur paths. >8 dB of degradation means multi-Fresnel-
|
||||
# zone disturbance — strong boost. 3-8 dB is a mild boost. <3 dB is noise.
|
||||
|
||||
test "nil degradation is a no-op" do
|
||||
assert Scorer.commercial_link_boost(75, nil) == 75
|
||||
end
|
||||
|
||||
test "<3 dB fade is noise and does not boost" do
|
||||
assert Scorer.commercial_link_boost(75, %{degradation_db: 2.0, n_links: 3}) == 75
|
||||
end
|
||||
|
||||
test "5 dB fade applies a mild boost" do
|
||||
boosted = Scorer.commercial_link_boost(75, %{degradation_db: 5.0, n_links: 3})
|
||||
assert boosted > 75
|
||||
assert boosted < 90
|
||||
end
|
||||
|
||||
test "12 dB fade applies a large boost capped at 100" do
|
||||
boosted = Scorer.commercial_link_boost(75, %{degradation_db: 12.0, n_links: 3})
|
||||
assert boosted > 85
|
||||
assert boosted <= 100
|
||||
end
|
||||
|
||||
test "single-link result still boosts (n_links >= 1)" do
|
||||
boosted = Scorer.commercial_link_boost(70, %{degradation_db: 6.0, n_links: 1})
|
||||
assert boosted > 70
|
||||
end
|
||||
|
||||
test "empty-links (n_links == 0) is a no-op" do
|
||||
assert Scorer.commercial_link_boost(75, %{degradation_db: 10.0, n_links: 0}) == 75
|
||||
end
|
||||
end
|
||||
|
||||
# ── score_sky/1 ──────────────────────────────────────────────────
|
||||
|
||||
describe "score_sky/1" do
|
||||
|
|
|
|||
|
|
@ -39,6 +39,72 @@ defmodule Microwaveprop.PropagationTest do
|
|||
assert is_integer(result.band_mhz)
|
||||
end)
|
||||
end
|
||||
|
||||
test "NEXRAD reflectivity drops the 24 GHz rain score when HRRR precip_mm is 0" do
|
||||
# Same profile, once without NEXRAD and once with a heavy-rain dBZ. The
|
||||
# 24 GHz rain-score factor must drop because NEXRAD can report rain that
|
||||
# HRRR's hourly precip accumulation hasn't caught yet.
|
||||
base_profile = %{
|
||||
surface_temp_c: 25.0,
|
||||
surface_dewpoint_c: 18.0,
|
||||
surface_pressure_mb: 1013.0,
|
||||
hpbl_m: 500.0,
|
||||
wind_u: 3.0,
|
||||
wind_v: 2.0,
|
||||
cloud_cover_pct: 15.0,
|
||||
precip_mm: 0.0,
|
||||
profile: [
|
||||
%{"pres" => 1000.0, "tmpc" => 25.0, "dwpc" => 18.0, "hght" => 100.0},
|
||||
%{"pres" => 950.0, "tmpc" => 19.0, "dwpc" => 10.0, "hght" => 600.0}
|
||||
]
|
||||
}
|
||||
|
||||
valid_time = ~U[2026-07-15 13:00:00Z]
|
||||
|
||||
without = Propagation.score_grid_point(base_profile, valid_time, 33.0, -97.0)
|
||||
|
||||
with_nexrad =
|
||||
base_profile
|
||||
|> Map.put(:nexrad_max_reflectivity_dbz, 45.0)
|
||||
|> Propagation.score_grid_point(valid_time, 33.0, -97.0)
|
||||
|
||||
rain_dry = Enum.find(without, &(&1.band_mhz == 24_000)).factors.rain
|
||||
rain_wet = Enum.find(with_nexrad, &(&1.band_mhz == 24_000)).factors.rain
|
||||
|
||||
assert rain_wet < rain_dry
|
||||
end
|
||||
|
||||
test "NEXRAD is ignored when HRRR precip_mm already reports heavier rain" do
|
||||
base_profile = %{
|
||||
surface_temp_c: 25.0,
|
||||
surface_dewpoint_c: 18.0,
|
||||
surface_pressure_mb: 1013.0,
|
||||
hpbl_m: 500.0,
|
||||
wind_u: 3.0,
|
||||
wind_v: 2.0,
|
||||
cloud_cover_pct: 15.0,
|
||||
# 30 mm/hr → far above the ~3 mm/hr NEXRAD gives for 30 dBZ, so HRRR wins.
|
||||
precip_mm: 30.0,
|
||||
profile: [
|
||||
%{"pres" => 1000.0, "tmpc" => 25.0, "dwpc" => 18.0, "hght" => 100.0},
|
||||
%{"pres" => 950.0, "tmpc" => 19.0, "dwpc" => 10.0, "hght" => 600.0}
|
||||
]
|
||||
}
|
||||
|
||||
valid_time = ~U[2026-07-15 13:00:00Z]
|
||||
|
||||
with_light_nexrad =
|
||||
base_profile
|
||||
|> Map.put(:nexrad_max_reflectivity_dbz, 30.0)
|
||||
|> Propagation.score_grid_point(valid_time, 33.0, -97.0)
|
||||
|
||||
without = Propagation.score_grid_point(base_profile, valid_time, 33.0, -97.0)
|
||||
|
||||
rain_no_nx = Enum.find(without, &(&1.band_mhz == 24_000)).factors.rain
|
||||
rain_with_nx = Enum.find(with_light_nexrad, &(&1.band_mhz == 24_000)).factors.rain
|
||||
|
||||
assert rain_with_nx == rain_no_nx
|
||||
end
|
||||
end
|
||||
|
||||
describe "upsert_scores/1" do
|
||||
|
|
|
|||
|
|
@ -29,6 +29,21 @@ defmodule Microwaveprop.Weather.Era5BatchClientTest do
|
|||
end
|
||||
|
||||
describe "fetch_month_into_db/1 idempotency" do
|
||||
# This describe block exercises the uncached path which would otherwise
|
||||
# hit the real CDS API if the developer has ERA5_CDS_API_KEY set via
|
||||
# direnv. Clear the env var for these tests so the expected "API key not
|
||||
# configured" short-circuit fires and the tests stay hermetic.
|
||||
setup do
|
||||
previous = System.get_env("ERA5_CDS_API_KEY")
|
||||
System.delete_env("ERA5_CDS_API_KEY")
|
||||
|
||||
on_exit(fn ->
|
||||
if previous, do: System.put_env("ERA5_CDS_API_KEY", previous)
|
||||
end)
|
||||
|
||||
:ok
|
||||
end
|
||||
|
||||
test "returns {:ok, 0} when the month-tile already has any profile" do
|
||||
# Seed one profile inside the 2014-03, tile (32, -98) window.
|
||||
%Era5Profile{}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue