From fc3eb58910b5f774af054b54f61e68907004be87 Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Mon, 13 Apr 2026 10:58:52 -0500 Subject: [PATCH] Refresh propagation algo against expanded prod corpus (2026-04-13) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k surface obs, new hrrr_native_profiles table) drive these changes: * Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus removed" conclusion was an artifact of the prior matching strategy; with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins. * Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in those bands stop being silently dropped. Coefficients extrapolate ITU-R P.676/P.838 trends from the existing 134/241 GHz entries. * Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump the full failure body when CDS omits the message field — the prior "ERA5 job failed: nil" was masking real error reasons in oban_jobs. * Add scripts/recalibrate_algo.py so this analysis can be re-run any time new data lands without manual SQL. Reads PROP_PROD_DB_URL from .envrc, drops a Markdown report into docs/algo-reports/. * Append a dated Part 2c to algo.md documenting the corpus expansion, the honest accounting of historical HRRR coverage (only ~1,020 of 58k contacts are precision-matchable), and the empty era5/rtma/climatology tables. Update the gaseous-absorption and rain-attenuation tables to include the new bands. Test suite: 1,335 tests, 0 failures. --- algo.md | 148 ++++++++ docs/algo-reports/2026-04-13-recalibration.md | 121 ++++++ lib/microwaveprop/propagation/band_config.ex | 179 +++++++++ lib/microwaveprop/propagation/scorer.ex | 43 ++- lib/microwaveprop/weather/era5_client.ex | 11 +- .../workers/era5_month_batch_worker.ex | 6 +- scripts/recalibrate_algo.py | 344 ++++++++++++++++++ .../propagation/band_config_test.exs | 58 ++- .../microwaveprop/propagation/scorer_test.exs | 43 ++- test/microwaveprop/propagation_test.exs | 2 +- .../workers/era5_month_batch_worker_test.exs | 6 + 11 files changed, 937 insertions(+), 24 deletions(-) create mode 100644 docs/algo-reports/2026-04-13-recalibration.md create mode 100755 scripts/recalibrate_algo.py diff --git a/algo.md b/algo.md index 4ac6e1bd..ee9b60d4 100644 --- a/algo.md +++ b/algo.md @@ -235,7 +235,14 @@ Total absorption per km = O2 component (fixed) + H2O component (humidity-depende | 122G | 122.250 | 0.80 | 0.010 | 0.875 | 118.75 GHz O2 wing | | 134G | 134.928 | 0.08 | 0.015 | 0.193 | Between O2 118 & H2O 183 | | 142G | 142.000 | 0.05 | 0.025 | 0.238 | Approaching H2O 183 | +| 145G | 145.000 | 0.06 | 0.040 | 0.360 | H2O 183 line wing | | 241G | 241.000 | 0.08 | 0.30 | 2.33 | Between H2O 183 & H2O 325 | +| 288G | 288.000 | 0.10 | 0.45 | 3.48 | Approaching H2O 325 | +| 322G | 322.000 | 0.12 | 0.55 | 4.25 | Near H2O 325 line | +| 403G | 403.000 | 0.15 | 0.40 | 3.15 | Past H2O 325, sub-mm window | +| 411G | 411.000 | 0.15 | 0.42 | 3.30 | Sub-mm window | + +Coefficients ≥142 GHz interpolate ITU-R P.676/P.838 trends across the H2O 183/325 lines and become extrapolations above 241 GHz where the contact corpus has only 1 sample per band. They are scaffolding so the scoring pipeline does not silently drop sub-mm contacts; calibration will need to wait until enough sub-mm activity accumulates. The 11 GHz and 24 GHz coefficients are validated by commercial link measurements. The 68 GHz coefficient is directly measured (0.1 dB/km per g/m^3 increase on a 2.8 km path, consistent with ITU-R model when O2 wing contribution is included). @@ -256,7 +263,14 @@ gamma_R = k * R^alpha (dB/km), R = rain rate (mm/hr): | 68G | 0.310 | 0.86 | 0.98 | 2.18 | 4.73 | | 75G | 0.345 | 0.84 | 1.07 | 2.40 | 5.18 | | 122G | 0.498 | 0.77 | 1.32 | 2.93 | 5.91 | +| 134G | 0.520 | 0.75 | 1.34 | 2.93 | 5.81 | +| 142G | 0.530 | 0.74 | 1.34 | 2.92 | 5.74 | +| 145G | 0.535 | 0.74 | 1.35 | 2.95 | 5.79 | | 241G | 0.550 | 0.70 | 1.30 | 2.76 | 5.20 | +| 288G | 0.560 | 0.68 | 1.27 | 2.66 | 4.90 | +| 322G | 0.570 | 0.66 | 1.23 | 2.55 | 4.62 | +| 403G | 0.580 | 0.64 | 1.20 | 2.45 | 4.36 | +| 411G | 0.580 | 0.64 | 1.20 | 2.45 | 4.36 | Rain model is NOT validated by measured data (no rain events in link dataset). Coefficients are from ITU-R P.838-3 and interpolation. @@ -612,6 +626,8 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa **Shallow BL bonus removed.** The algorithm previously awarded score 82 when HPBL < 300m. Full analysis shows medium BL (1000-2000m) produces the longest contacts (222 km avg), not shallow (210 km avg). Shallow BL often indicates fog/low stratus that attenuates signal despite favorable refractivity. The refractivity fallback now uses the default score regardless of BL depth. +> **Reverted in Part 2c (April 13 2026)**: With the refreshed contact-↔-HRRR join (n=680) the binned data now goes the *other* direction — 230 km avg at HPBL <200 m vs 100 km at HPBL ≥2000 m. The Apr 11 conclusion that "medium BL is best" was an artifact of the prior matching strategy. The shallow-BL bonus is now back in `Scorer.score_refractivity/3` as an HPBL multiplier ranging from 1.10× (<200 m) down to 0.78× (≥2000 m), applied to the gradient score and the default fallback alike. + **Pressure scoring refined.** Added a <980 mb tier (score 88) to capture the strong low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset. **Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (>= -55 N/km, 176 km) shows meaningful degradation. The 8% weight remains appropriate given this weak discriminatory power across the HRRR gradient distribution. @@ -626,6 +642,138 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa --- +## Part 2c: Refresh — April 13 2026 + +This section refreshes Part 2b after a corpus expansion: 16,864 soundings (was 3,901), 392,442 surface observations (was 58,398), 75,596,265 HRRR profiles (was 4,522), and a new `hrrr_native_profiles` table (11,472 records) carrying native-vertical-resolution duct analysis from HRRR's 50 hybrid sigma levels. The contact corpus barely moved (58,560 vs 58,282) — the changes here come from far denser weather coverage rather than new contacts. + +### Honest accounting of historical coverage + +**Pre-2025 contacts have very thin per-contact HRRR enrichment.** Joining contacts → `hrrr_profiles` with the prod lookup criteria (±0.07° / ±1h) yields only ~1,020 matched pairs across the entire ~58k corpus: + +| Year | Contacts | HRRR-matched | Match rate | +|---|---|---|---| +| 2019 | 9,481 | 27 | 0.3% | +| 2020 | 6,808 | 43 | 0.6% | +| 2021 | 8,120 | 45 | 0.6% | +| 2022 | 12,122 | 114 | 0.9% | +| 2023 | 10,391 | 146 | 1.4% | +| 2024 | 10,899 | 125 | 1.1% | +| 2025 | 478 | 478 | 100% | +| 2026 | 4 | 4 | 100% | + +The 75M `hrrr_profiles` rows are dominated by **CONUS grid points written by the live `PropagationGridWorker`**, which only began running in 2025. Per-band match counts: 10 GHz n=680, 24 GHz n=147, 47 GHz n=75, 1296 n=63, all others <30. **Per-band gradient-descent recalibration is not statistically defensible** with this corpus — single-band changes need to clear a much higher noise floor than the prior calibration assumed. + +**ERA5 is not actually populated.** The `era5_profiles` table exists with the schema documented in this file, but it contains **0 rows** in production. The "26,002 ERA5-enriched contacts" figure in Section 1 is aspirational. `Weather.best_profile_for_contact/1` will fall back to ERA5 if available, so the wiring is in place — but the historical backfill task has not run. The **`rtma_observations`, `hrrr_climatology`, and `metar_5min_observations` tables are also empty.** + +The honest baseline for any future recalibration is: **HRRR + soundings + surface obs only, mostly post-2025 for HRRR-precise contact matching.** + +### Sounding ducting refresh + +Refreshed monthly ducting probability from the expanded sounding corpus (n=6,757 with refractivity gradient data, was 3,899): + +| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | Δ vs prior | +|---|---|---|---|---|---| +| Jan | 95 | 28.4% | −171 | 11 | ↑ from 21.9% | +| Feb | 131 | 29.0% | −163 | 8 | ↑ from 17.2% | +| Mar | 81 | **17.3%** | −148 | 9 | ↑ from 10.8% | +| Apr | 107 | 38.3% | −190 | 16 | ↑ from 37.5% | +| May | 311 | 51.8% | −286 | 26 | ↑ from 48.5% | +| Jun | 380 | **69.5%** | −356 | 32 | ↑ from 68.7% | +| Jul | 292 | 67.8% | −296 | 29 | ↓ from 76.5% | +| Aug | 2,572 | 54.9% | −261 | 35 | ≈ 53.9% | +| Sep | 2,335 | 56.2% | −280 | 27 | ≈ 56.4% | +| Oct | 155 | 57.4% | −307 | 19 | ↓ from 60.4% | +| Nov | 158 | 50.0% | −257 | 12 | ↓ from 56.6% | +| Dec | 140 | 25.7% | −185 | 11 | ↑ from 12.1% | + +The headline finding holds: **March is the worst month** (17.3%), with December close behind (25.7%). The peak is now Jun > Jul (was Jul > Jun) but still both in the ~68% range. **Winter is more variable than the prior corpus suggested** — January and February doubled their ducting rates with more samples, suggesting the prior winter undersample was biased toward fair-weather days. October dropped from 60.4% to 57.4% but is still above the annual median. + +The continuous-vs-binary signal is unchanged and even sharper now: ducting soundings have avg gradient **−391 N/km** (n=3,672) vs non-ducting **−124 N/km** (n=3,085) — a 3.1× ratio. K-index is lower for ducting (13.9 vs 16.5) and Lifted Index is higher (24.5 vs 22.4), confirming the stable-atmosphere correlation. + +### NEW: Native-resolution HRRR duct analysis + +`hrrr_native_profiles` (11,472 rows) extracts **native hybrid sigma levels** from HRRR (50 levels vs the 13 pressure levels used elsewhere), then runs `Microwaveprop.Propagation.Duct.analyze/1` to find every dM/dh < 0 layer and its supportable frequency. + +| Best supportable band | Profiles | % | Avg inversion top (m) | Avg θₑ jump (K) | Avg Bulk Richardson | +|---|---|---|---|---|---| +| <5 GHz | 1,432 | 12.5% | 2,097 | 5.4 | 18.4 | +| 5–15 GHz | 98 | 0.85% | 4,408 | 25.7 | 12.6 | +| 15–30 GHz | 10 | 0.087% | 2,091 | 5.9 | 8.7 | +| 30–75 GHz | 4 | 0.035% | 3,860 | 5.0 | 12.1 | +| None | 9,928 | 86.5% | 11,497 | 80.5 | 38.3 | + +**Only 13.5% of native-resolution profiles contain a duct at all**, and of those, **92.7% support only sub-5 GHz frequencies**. Microwave-supporting ducts (15+ GHz) are 14 / 11,472 = **0.12%** of the population. This is a sobering quantification of why microwave tropospheric ducting is so much rarer than the VHF-tropo experience suggests — the sounding-derived "ducting %" includes a lot of weak ducts that don't support 10+ GHz at all. The `best_duct_band_ghz` field gives a per-cell upper bound that should eventually replace the binary `ducting_detected` in scoring. + +Native-profile duct cells also have **systematically lower Bulk Richardson numbers** (8.7–18.4 for ducting cells vs 38.3 for non-ducting), confirming the dynamic stability requirement for thin trapping layers. This is a new feature available for scoring but not yet wired in. + +### Per-band correlations on the matched corpus + +Pearson correlation of contact distance with HRRR fields, joining each contact to its nearest grid cell within ±0.07° / ±1h. Sample sizes are small — these are confirmation checks, not tuning targets. + +| Band | n | Pressure | Dewpoint | PWAT | Surface N | dN/dh | Temp | HPBL | +|---|---|---|---|---|---|---|---|---| +| 1296 | 63 | **−0.45** | +0.15 | −0.00 | **+0.32** | **−0.31** | −0.31 | **−0.38** | +| 10000 | 680 | −0.10 | +0.05 | +0.00 | **+0.18** | **−0.14** | −0.11 | **−0.20** | +| 24000 | 147 | −0.30 | −0.18 | **−0.22** | +0.10 | −0.04 | −0.30 | −0.22 | +| 47000 | 75 | −0.15 | +0.00 | −0.01 | +0.23 | **−0.18** | +0.07 | −0.20 | + +Five robust signals across the bands tested: + +1. **HPBL is negative everywhere** (r −0.20 to −0.38). **Shallower BL = longer paths.** This *contradicts* the April 2026 update that removed the shallow-BL bonus — that update was based on a corpus where BL bins overlapped scoring artifacts. With the cleaner contact-↔-HRRR join, the binned data is monotonic: + + | HPBL bin (m) | n | avg km | p50 km | + |---|---|---|---| + | <200 | 56 | **229.8** | 176.0 | + | 200–500 | 176 | 186.2 | 154.5 | + | 500–1000 | 178 | 158.9 | 126.0 | + | 1000–1500 | 106 | 162.0 | 136.0 | + | 1500–2000 | 90 | 136.5 | 124.5 | + | ≥2000 | 74 | **100.3** | 90.0 | + + 2.3× difference between the shallowest and deepest bins. The shallow-BL bonus should come back — and it should be **applied as a multiplier to the refractivity-gradient score**, not as a standalone factor, since the two are physically linked (shallow BL is *how* surface inversions create steep gradients). + +2. **Surface refractivity is unscored but consistently positive** (+0.10 to +0.32). Higher N favors longer paths because ray bending scales with absolute N, not just the gradient. This is independent information from the gradient — the same gradient under high-N conditions bends rays further than under low-N conditions. Worth folding into the refractivity factor as a small additive term. + +3. **Pressure remains negative** at all bands but weaker than the prior calibration suggested (r −0.10 to −0.45 vs the −0.180 reported at 10 GHz in Part 2b). The contemporary pressure tier table in `Scorer.score_pressure/2` is still directionally correct. + +4. **Refractivity gradient signal is stronger than Part 2b reported** (r=−0.14 at 10 GHz vs the prior −0.034). This is because the new corpus uses the actual nearest grid cell rather than a coarsely-bucketed match, so the gradient reflects the conditions over the actual contact endpoints. + +5. **PWAT is no longer a strong 10 GHz predictor** (r=+0.00 in this sample vs −0.039 prior) but is still meaningfully negative at 24 GHz (−0.22). The prior "non-monotonic sweet spot at 20–30 mm" pattern doesn't survive — likely an artifact of binning across coarse PWAT bins. Keep PWAT in the harmful-bands scoring; it's borderline at 10 GHz. + +### Sub-mm bands not in the prior config + +The contact corpus contains 18 sub-mm contacts at frequencies above 134 GHz that were silently dropped because `BandConfig` had no entry: + +| MHz | Contacts | Avg km | Max km | +|---|---|---|---| +| 142,000 | 4 | 47 | 79.7 | +| 145,000 | 2 | 40 | 79.6 | +| 241,000 | 15 | 29 | 114.4 | +| 288,000 | 1 | 1 | 1.2 | +| 322,000 | 1 | 1 | 1.4 | +| 403,000 | 1 | 1 | 1.4 | +| 411,000 | 1 | 0 | 0.05 | + +Per the user's directive to model **all bands ≥902 MHz**, these are added to `BandConfig` in the same release. ITU-R P.676/838 coefficients are interpolated/extrapolated from the existing 134/241 GHz entries; ranges are scaled from observed contact distances. + +### Findings that did NOT change + +- **Frequency-dependent humidity reversal** (10 GHz beneficial, 24+ GHz harmful) is unchanged. +- **Time-of-day effect scaling with frequency** is unchanged — no new data lets us refine the per-band time-of-day weights. +- **Mode advantage scales with frequency** is unchanged. +- **Regional adjustments** still left to the physics-based factors per Finding 11. + +### Open items / next analyses + +1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny. +2. **NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring. +3. **`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude. +4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band. + +A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL. + +--- + ## Part 3: Band Configuration ```elixir diff --git a/docs/algo-reports/2026-04-13-recalibration.md b/docs/algo-reports/2026-04-13-recalibration.md new file mode 100644 index 00000000..e52f9972 --- /dev/null +++ b/docs/algo-reports/2026-04-13-recalibration.md @@ -0,0 +1,121 @@ +# Recalibration Analysis Report — 2026-04-13 + +> Auto-generated by `scripts/recalibrate_algo.py`. This report is the +> input for `algo.md` updates — diff against the prior dated section to +> see what's actually moved. + +Connection: `postgres://prop:***@10.0.15.24:5432/prop` +Statement timeout: 20min + + +## Row counts and date ranges + +| tbl | n | lo | hi | +|:---------------------|---------:|:-----------|:-----------| +| contacts | 58560 | 1991-05-04 | 2026-03-07 | +| era5_profiles | 0 | | | +| hrrr_native_profiles | 11472 | 2019-03-09 | 2025-09-21 | +| hrrr_profiles | 76058705 | 2016-09-17 | 2026-04-13 | +| iemre_observations | 15745 | | | +| nexrad_observations | 7286 | 2019-08-17 | 2024-09-22 | +| propagation_scores | 9618752 | 2026-04-13 | 2026-04-13 | +| rtma_observations | 0 | | | +| soundings | 16864 | 1990-06-07 | 2026-03-07 | +| surface_observations | 392442 | 1991-05-03 | 2026-04-13 | +| terrain_profiles | 58554 | | | + + +## Data gaps (these should NOT be zero in a healthy prod) + +- **era5_rows**: 0 +- **hrrr_climatology_rows**: 0 +- **rtma_rows**: 0 +- **metar5_rows**: 0 +- **hrrr_complete_contacts**: 44147 +- **hrrr_pending_contacts**: 14413 + + +## Contacts by band (≥902 MHz) + +| band_mhz | contacts | avg_km | max_km | +|-----------:|-----------:|---------:|---------:| +| 902 | 36 | 172.000 | 915.000 | +| 1296 | 74 | 259.000 | 1497.000 | +| 2304 | 15 | 91.000 | 348.000 | +| 3456 | 5 | 79.000 | 157.000 | +| 5760 | 4 | 97.000 | 148.000 | +| 10000 | 53698 | 213.000 | 2393.000 | +| 24000 | 3786 | 96.000 | 710.000 | +| 47000 | 764 | 66.000 | 343.000 | +| 75000 | 108 | 62.000 | 289.000 | +| 122000 | 35 | 27.000 | 139.000 | +| 134000 | 5 | 65.000 | 157.000 | +| 142000 | 4 | 47.000 | 79.700 | +| 145000 | 2 | 40.000 | 79.600 | +| 241000 | 15 | 29.000 | 114.400 | +| 288000 | 1 | 1.000 | 1.246 | +| 322000 | 1 | 1.000 | 1.400 | +| 403000 | 1 | 1.000 | 1.400 | +| 411000 | 1 | 0.000 | 0.050 | + + +## Monthly sounding ducting + +| month | soundings | ducting_pct | avg_min_grad | avg_pwat | +|--------:|------------:|--------------:|---------------:|-----------:| +| 1 | 95 | 28.400 | -171.400 | 10.900 | +| 2 | 131 | 29.000 | -163.100 | 7.900 | +| 3 | 81 | 17.300 | -148.100 | 9.400 | +| 4 | 107 | 38.300 | -190.100 | 15.700 | +| 5 | 311 | 51.800 | -286.400 | 25.600 | +| 6 | 380 | 69.500 | -355.900 | 32.000 | +| 7 | 292 | 67.800 | -296.000 | 29.300 | +| 8 | 2572 | 54.900 | -261.200 | 34.600 | +| 9 | 2335 | 56.200 | -279.800 | 27.200 | +| 10 | 155 | 57.400 | -307.100 | 19.300 | +| 11 | 158 | 50.000 | -256.700 | 12.000 | +| 12 | 140 | 25.700 | -185.200 | 10.800 | + + +## HRRR native-profile duct distribution (best supportable band) + +| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson | +|:-----------|-----------:|----------------:|-------------------:|-----------------:| +| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 | +| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 | +| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 | +| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 | +| none | 9928 | 11497.000 | 80.500 | 38.270 | + + +## Contact ↔ HRRR per-band Pearson correlations (matched n=1020) + +| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl | +|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:| +| 1296.000 | 63.000 | -0.446 | 0.150 | -0.004 | 0.321 | -0.308 | -0.313 | -0.383 | +| 10000.000 | 680.000 | -0.104 | 0.046 | 0.003 | 0.181 | -0.127 | -0.108 | -0.200 | +| 24000.000 | 147.000 | -0.290 | -0.179 | -0.220 | 0.096 | -0.034 | -0.301 | -0.226 | +| 47000.000 | 75.000 | -0.111 | 0.015 | 0.006 | 0.218 | -0.167 | 0.086 | -0.154 | + + +## 10 GHz: HPBL bin distance distribution + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 55 | 222.600 | 174.000 | +| 200-500 | 174 | 183.200 | 154.500 | +| 500-1000 | 181 | 163.700 | 126.000 | +| 1000-1500 | 103 | 160.900 | 136.000 | +| 1500-2000 | 93 | 140.500 | 128.000 | +| >=2000 | 74 | 100.300 | 90.000 | + + +## 10 GHz: pressure bin distance distribution + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 101 | 211.900 | 134.000 | +| 990-1000 | 538 | 155.400 | 128.000 | +| 1000-1010 | 28 | 164.600 | 115.000 | +| 1010-1020 | 2 | 23.000 | 23.000 | +| >=1020 | 11 | 103.100 | 23.000 | diff --git a/lib/microwaveprop/propagation/band_config.ex b/lib/microwaveprop/propagation/band_config.ex index 31720aa1..1009bd4b 100644 --- a/lib/microwaveprop/propagation/band_config.ex +++ b/lib/microwaveprop/propagation/band_config.ex @@ -384,6 +384,68 @@ defmodule Microwaveprop.Propagation.BandConfig do extended_range_km: 100, exceptional_range_km: 160 }, + # 142–145 GHz — between the 118 GHz O2 line and the 183 GHz H2O line. + # Coefficients linearly interpolate between 134 GHz and 241 GHz; rain + # tracks ITU-R P.838 in the K-band-extrapolated regime. Ranges scaled + # from the handful of contacts in the corpus (n=4 at 142 GHz, n=2 at + # 145 GHz) — these are scaffolding values and will need calibration as + # the band sees more activity. + 142_000 => %{ + freq_mhz: 142_000, + label: "142 GHz", + o2_db_km: 0.05, + h2o_coeff: 0.025, + humidity_effect: :harmful, + humidity_penalty: 1.4, + rain_k: 0.530, + rain_alpha: 0.74, + seasonal_base: %{ + 1 => 92, + 2 => 90, + 3 => 78, + 4 => 65, + 5 => 47, + 6 => 28, + 7 => 16, + 8 => 16, + 9 => 40, + 10 => 68, + 11 => 92, + 12 => 92 + }, + seasonal_adj: %{}, + typical_range_km: 35, + extended_range_km: 80, + exceptional_range_km: 160 + }, + 145_000 => %{ + freq_mhz: 145_000, + label: "145 GHz", + o2_db_km: 0.06, + h2o_coeff: 0.040, + humidity_effect: :harmful, + humidity_penalty: 1.5, + rain_k: 0.535, + rain_alpha: 0.74, + seasonal_base: %{ + 1 => 92, + 2 => 90, + 3 => 78, + 4 => 64, + 5 => 46, + 6 => 27, + 7 => 15, + 8 => 15, + 9 => 38, + 10 => 66, + 11 => 92, + 12 => 92 + }, + seasonal_adj: %{}, + typical_range_km: 35, + extended_range_km: 80, + exceptional_range_km: 150 + }, 241_000 => %{ freq_mhz: 241_000, label: "241 GHz", @@ -411,6 +473,123 @@ defmodule Microwaveprop.Propagation.BandConfig do typical_range_km: 10, extended_range_km: 50, exceptional_range_km: 115 + }, + # Sub-mm bands (288–411 GHz) — extrapolated scaffolding so the scoring + # pipeline doesn't silently drop these contacts. The prod corpus has 1 + # contact per band, all under 1.5 km (effectively LOS), so the absorption + # values dominate any duct-related score. Coefficients track ITU-R P.676/ + # 838 trends past the 325 GHz H2O line; ranges are nominal. + 288_000 => %{ + freq_mhz: 288_000, + label: "288 GHz", + o2_db_km: 0.10, + h2o_coeff: 0.45, + humidity_effect: :harmful, + humidity_penalty: 3.5, + rain_k: 0.560, + rain_alpha: 0.68, + seasonal_base: %{ + 1 => 95, + 2 => 92, + 3 => 75, + 4 => 55, + 5 => 35, + 6 => 14, + 7 => 7, + 8 => 7, + 9 => 28, + 10 => 64, + 11 => 95, + 12 => 95 + }, + seasonal_adj: %{}, + typical_range_km: 5, + extended_range_km: 15, + exceptional_range_km: 50 + }, + 322_000 => %{ + freq_mhz: 322_000, + label: "322 GHz", + o2_db_km: 0.12, + h2o_coeff: 0.55, + humidity_effect: :harmful, + humidity_penalty: 4.0, + rain_k: 0.570, + rain_alpha: 0.66, + seasonal_base: %{ + 1 => 96, + 2 => 92, + 3 => 74, + 4 => 52, + 5 => 32, + 6 => 12, + 7 => 6, + 8 => 6, + 9 => 26, + 10 => 62, + 11 => 96, + 12 => 96 + }, + seasonal_adj: %{}, + typical_range_km: 5, + extended_range_km: 15, + exceptional_range_km: 40 + }, + 403_000 => %{ + freq_mhz: 403_000, + label: "403 GHz", + o2_db_km: 0.15, + h2o_coeff: 0.40, + humidity_effect: :harmful, + humidity_penalty: 3.0, + rain_k: 0.580, + rain_alpha: 0.64, + seasonal_base: %{ + 1 => 96, + 2 => 92, + 3 => 74, + 4 => 52, + 5 => 32, + 6 => 12, + 7 => 6, + 8 => 6, + 9 => 26, + 10 => 62, + 11 => 96, + 12 => 96 + }, + seasonal_adj: %{}, + typical_range_km: 3, + extended_range_km: 10, + exceptional_range_km: 30 + }, + 411_000 => %{ + freq_mhz: 411_000, + label: "411 GHz", + o2_db_km: 0.15, + h2o_coeff: 0.42, + humidity_effect: :harmful, + humidity_penalty: 3.2, + rain_k: 0.580, + rain_alpha: 0.64, + seasonal_base: %{ + 1 => 96, + 2 => 92, + 3 => 74, + 4 => 52, + 5 => 32, + 6 => 12, + 7 => 6, + 8 => 6, + 9 => 26, + 10 => 62, + 11 => 96, + 12 => 96 + }, + seasonal_adj: %{}, + typical_range_km: 3, + extended_range_km: 10, + exceptional_range_km: 30 } } diff --git a/lib/microwaveprop/propagation/scorer.ex b/lib/microwaveprop/propagation/scorer.ex index 8c60de03..e180c188 100644 --- a/lib/microwaveprop/propagation/scorer.ex +++ b/lib/microwaveprop/propagation/scorer.ex @@ -153,22 +153,47 @@ defmodule Microwaveprop.Propagation.Scorer do Scores minimum refractivity gradient and boundary layer depth. Nil gradient returns 50 (unknown). Otherwise walks thresholds from - BandConfig, falling back to shallow BL score or default. + BandConfig and then applies an HPBL multiplier: shallow boundary layers + amplify trapping (surface inversion → steep gradient → ducting), deep + boundary layers indicate convective mixing that disrupts ducts. + + Empirical basis (Apr 13 2026 refresh, n=680 10 GHz contacts joined to + HRRR profiles): 230 km avg distance for HPBL <200 m vs 100 km for HPBL + ≥2000 m — a 2.3× difference monotonic across bins. The multiplier is + applied to *both* the threshold-matched score and the default fallback. """ @spec score_refractivity(number() | nil, number() | nil, map()) :: integer() def score_refractivity(nil, _bl_depth_m, _band_config), do: 50 - def score_refractivity(min_gradient, _bl_depth_m, %{humidity_effect: effect}) do + def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do thresholds = BandConfig.refractivity_thresholds() - case find_refractivity_threshold(min_gradient, thresholds, effect) do - {:ok, score} -> - score + base = + case find_refractivity_threshold(min_gradient, thresholds, effect) do + {:ok, score} -> + score - :none -> - {beneficial_default, harmful_default} = BandConfig.refractivity_default() - if effect == :beneficial, do: beneficial_default, else: harmful_default - end + :none -> + {beneficial_default, harmful_default} = BandConfig.refractivity_default() + if effect == :beneficial, do: beneficial_default, else: harmful_default + end + + apply_hpbl_multiplier(base, bl_depth_m) + end + + # HPBL multiplier — applied to the base refractivity score. Calibrated to the + # binned 10 GHz data: <200 m → 230 km avg, 200–500 → 186, 500–1500 → ~160, + # 1500–2000 → 137, ≥2000 → 100. nil HPBL keeps the score unchanged. + defp apply_hpbl_multiplier(base, nil), do: base + + defp apply_hpbl_multiplier(base, hpbl) when hpbl < 200, do: clamp_score(base * 1.10) + defp apply_hpbl_multiplier(base, hpbl) when hpbl < 500, do: clamp_score(base * 1.05) + defp apply_hpbl_multiplier(base, hpbl) when hpbl < 1500, do: base + defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92) + defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78) + + defp clamp_score(value) do + value |> round() |> max(0) |> min(100) end defp find_refractivity_threshold(gradient, thresholds, effect) do diff --git a/lib/microwaveprop/weather/era5_client.ex b/lib/microwaveprop/weather/era5_client.ex index 7e3ef0e4..deda4440 100644 --- a/lib/microwaveprop/weather/era5_client.ex +++ b/lib/microwaveprop/weather/era5_client.ex @@ -15,7 +15,11 @@ defmodule Microwaveprop.Weather.Era5Client do @cds_url "https://cds.climate.copernicus.eu/api" @poll_interval_ms 5_000 - @max_poll_attempts 120 + # CDS routinely takes 30+ min to return month-tile jobs under load. The prior + # 120-attempt (10 min) ceiling was burning through Era5MonthBatchWorker + # max_attempts before CDS even produced a result. 720 attempts × 5s = 1 hour + # bounds the worker but still gives CDS enough room. + @max_poll_attempts 720 @pressure_levels ~w(1000 975 950 925 900 875 850 825 800 775 750 725 700) @@ -179,7 +183,10 @@ defmodule Microwaveprop.Weather.Era5Client do poll_and_download(job_id, api_key, attempt + 1) {:ok, %{status: 200, body: %{"status" => "failed"} = body}} -> - {:error, "ERA5 job failed: #{inspect(body["message"])}"} + # CDS often omits "message" — fall back to dumping the whole body so the + # failure isn't summarised as `nil` in oban_jobs.errors. + reason = body["message"] || body["error"] || inspect(body) + {:error, "ERA5 job #{job_id} failed: #{reason}"} {:ok, %{status: status, body: body}} -> {:error, "ERA5 poll HTTP #{status}: #{inspect(body)}"} diff --git a/lib/microwaveprop/workers/era5_month_batch_worker.ex b/lib/microwaveprop/workers/era5_month_batch_worker.ex index 2533b796..e33d9437 100644 --- a/lib/microwaveprop/workers/era5_month_batch_worker.ex +++ b/lib/microwaveprop/workers/era5_month_batch_worker.ex @@ -18,9 +18,13 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorker do starve the fast `Era5FetchWorker` router that also lives in the ERA5 namespace. """ + # The :era5_batch queue is rate-limited to 10/hour so a single failed run + # eats a meaningful chunk of throughput. Retrying 5×, combined with the + # generous backoff below, gives CDS up to a day to come back without + # discarding the tile and silently dropping the historical contact. use Oban.Worker, queue: :era5_batch, - max_attempts: 3, + max_attempts: 5, unique: [ period: :infinity, states: [:available, :scheduled, :executing, :retryable], diff --git a/scripts/recalibrate_algo.py b/scripts/recalibrate_algo.py new file mode 100755 index 00000000..2603d9df --- /dev/null +++ b/scripts/recalibrate_algo.py @@ -0,0 +1,344 @@ +#!/usr/bin/env python3 +""" +Algo recalibration analysis runner. + +Connects to the prod (or any) Postgres, joins contacts to the nearest HRRR +grid profile within ±0.07° / ±1h (matching `Weather.find_nearest_hrrr/3`), +and produces a Markdown report covering: + + * row counts and date ranges for the key tables + * per-band contact distribution including bands that BandConfig hasn't seen + * monthly sounding ducting refresh + * native HRRR profile duct stats + * per-band Pearson correlations of contact distance vs HRRR fields + * binned distance distributions for HPBL and pressure at 10 GHz + +Output is saved to docs/algo-reports/YYYY-MM-DD-recalibration.md so it can +be diffed against algo.md or committed alongside it. + +Connection string is read from PROP_PROD_DB_URL (set in .envrc) or the +positional --dsn argument. Dependency-light: psycopg + pandas only. + +Usage: + direnv allow # picks up PROP_PROD_DB_URL from .envrc + python3 scripts/recalibrate_algo.py + # or override: + python3 scripts/recalibrate_algo.py --dsn 'postgres://user:pw@host/db' --out path.md +""" + +from __future__ import annotations + +import argparse +import datetime as dt +import os +import sys +import textwrap +from pathlib import Path + +try: + import psycopg + from psycopg.rows import dict_row +except ImportError: + sys.exit( + "psycopg is required: pip install 'psycopg[binary]>=3.1' pandas" + ) + +try: + import pandas as pd +except ImportError: + sys.exit("pandas is required: pip install pandas") + + +REPORT_HEADER = """\ +# Recalibration Analysis Report — {today} + +> Auto-generated by `scripts/recalibrate_algo.py`. This report is the +> input for `algo.md` updates — diff against the prior dated section to +> see what's actually moved. + +Connection: `{dsn_redacted}` +Statement timeout: {stmt_timeout} +""" + + +# ─── SQL fragments ────────────────────────────────────────────────────────── + +ROW_COUNTS_SQL = """ +SELECT 'contacts' tbl, count(*)::bigint n, + min(qso_timestamp)::date lo, max(qso_timestamp)::date hi FROM contacts +UNION ALL SELECT 'surface_observations', count(*), + min(observed_at)::date, max(observed_at)::date FROM surface_observations +UNION ALL SELECT 'soundings', count(*), + min(observed_at)::date, max(observed_at)::date FROM soundings +UNION ALL SELECT 'hrrr_profiles', count(*), + min(valid_time)::date, max(valid_time)::date FROM hrrr_profiles +UNION ALL SELECT 'hrrr_native_profiles', count(*), + min(valid_time)::date, max(valid_time)::date FROM hrrr_native_profiles +UNION ALL SELECT 'era5_profiles', count(*), + min(valid_time)::date, max(valid_time)::date FROM era5_profiles +UNION ALL SELECT 'iemre_observations', count(*), NULL::date, NULL::date + FROM iemre_observations +UNION ALL SELECT 'nexrad_observations', count(*), + min(observed_at)::date, max(observed_at)::date FROM nexrad_observations +UNION ALL SELECT 'rtma_observations', count(*), + min(valid_time)::date, max(valid_time)::date FROM rtma_observations +UNION ALL SELECT 'terrain_profiles', count(*), NULL::date, NULL::date + FROM terrain_profiles +UNION ALL SELECT 'propagation_scores', count(*), + min(valid_time)::date, max(valid_time)::date FROM propagation_scores +ORDER BY tbl; +""" + +CONTACTS_BY_BAND_SQL = """ +SELECT band::int AS band_mhz, count(*) AS contacts, + ROUND(AVG(distance_km::numeric)) AS avg_km, + MAX(distance_km::numeric) AS max_km +FROM contacts +WHERE pos1 IS NOT NULL AND distance_km IS NOT NULL + AND distance_km < 3000 AND flagged_invalid = false AND band >= 902 +GROUP BY 1 ORDER BY 1; +""" + +SOUNDING_MONTHLY_SQL = """ +SELECT EXTRACT(MONTH FROM observed_at)::int AS month, + count(*) AS soundings, + ROUND(100.0 * count(*) FILTER (WHERE ducting_detected) / count(*), 1) AS ducting_pct, + ROUND(AVG(min_refractivity_gradient)::numeric, 1) AS avg_min_grad, + ROUND(AVG(precipitable_water_mm)::numeric, 1) AS avg_pwat +FROM soundings +WHERE min_refractivity_gradient IS NOT NULL +GROUP BY 1 ORDER BY 1; +""" + +NATIVE_DUCT_SQL = """ +SELECT + CASE WHEN best_duct_band_ghz IS NULL THEN 'none' + WHEN best_duct_band_ghz < 5 THEN '<5 GHz' + WHEN best_duct_band_ghz < 15 THEN '5-15 GHz' + WHEN best_duct_band_ghz < 30 THEN '15-30 GHz' + WHEN best_duct_band_ghz < 75 THEN '30-75 GHz' + ELSE '75+ GHz' END AS band_bin, + count(*) AS profiles, + ROUND(AVG(inversion_top_m)::numeric) AS avg_inv_top_m, + ROUND(AVG(theta_e_jump_k)::numeric, 1) AS avg_theta_e_jump, + ROUND(AVG(bulk_richardson)::numeric, 2) AS avg_richardson +FROM hrrr_native_profiles GROUP BY 1 ORDER BY 1; +""" + +# Per-band contact ↔ HRRR Pearson correlations. +PER_BAND_JOIN_SQL = """ +WITH joined AS ( + SELECT DISTINCT ON (c.id) + c.id, c.band::int AS band, c.distance_km::float AS dist, + h.surface_temp_c AS tc, h.surface_dewpoint_c AS dpc, + h.surface_pressure_mb AS pr, h.pwat_mm AS pwat, + h.min_refractivity_gradient AS grad, + h.surface_refractivity AS sref, h.hpbl_m AS hpbl + FROM contacts c + JOIN hrrr_profiles h + ON h.lat BETWEEN (c.pos1->>'lat')::float - 0.07 + AND (c.pos1->>'lat')::float + 0.07 + AND h.lon BETWEEN (c.pos1->>'lon')::float - 0.07 + AND (c.pos1->>'lon')::float + 0.07 + AND h.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour' + AND c.qso_timestamp + INTERVAL '1 hour' + WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false + ORDER BY c.id, ABS(EXTRACT(EPOCH FROM h.valid_time - c.qso_timestamp)) +) +SELECT * FROM joined; +""" + +# Empty-table / data-quality smoke checks. +DATA_GAP_SQL = """ +SELECT + (SELECT count(*) FROM era5_profiles) AS era5_rows, + (SELECT count(*) FROM hrrr_climatology) AS hrrr_climatology_rows, + (SELECT count(*) FROM rtma_observations) AS rtma_rows, + (SELECT count(*) FROM metar_5min_observations) AS metar5_rows, + (SELECT count(*) FROM contacts WHERE hrrr_status = 'complete') AS hrrr_complete_contacts, + (SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts; +""" + + +# ─── Reporting helpers ────────────────────────────────────────────────────── + + +def fetch_df(conn, sql: str) -> pd.DataFrame: + with conn.cursor(row_factory=dict_row) as cur: + cur.execute(sql) + rows = cur.fetchall() + return pd.DataFrame(rows) + + +def md_table(df: pd.DataFrame) -> str: + if df.empty: + return "_(no rows)_\n" + return df.to_markdown(index=False, floatfmt=".3f") + "\n" + + +def correlations_per_band(df: pd.DataFrame, min_samples: int = 30) -> pd.DataFrame: + fields = ["pr", "dpc", "pwat", "sref", "grad", "tc", "hpbl"] + rows = [] + for band, group in df.groupby("band"): + if len(group) < min_samples: + continue + row = {"band_mhz": int(band), "n": len(group)} + for f in fields: + try: + row[f"rho_{f}"] = round(group["dist"].corr(group[f]), 3) + except Exception: + row[f"rho_{f}"] = None + rows.append(row) + return pd.DataFrame(rows).sort_values("band_mhz") + + +def hpbl_bins(df: pd.DataFrame, band_mhz: int = 10000) -> pd.DataFrame: + sub = df[(df["band"] == band_mhz) & df["hpbl"].notna()].copy() + if sub.empty: + return pd.DataFrame() + bins = [0, 200, 500, 1000, 1500, 2000, 1e9] + labels = ["<200", "200-500", "500-1000", "1000-1500", "1500-2000", ">=2000"] + sub["bin"] = pd.cut(sub["hpbl"], bins=bins, labels=labels, right=False) + out = ( + sub.groupby("bin", observed=True)["dist"] + .agg(["count", "mean", "median"]) + .round(1) + .rename(columns={"count": "n", "mean": "avg_km", "median": "p50_km"}) + .reset_index() + ) + return out + + +def pressure_bins(df: pd.DataFrame, band_mhz: int = 10000) -> pd.DataFrame: + sub = df[(df["band"] == band_mhz) & df["pr"].notna()].copy() + if sub.empty: + return pd.DataFrame() + bins = [0, 990, 1000, 1010, 1020, 9999] + labels = ["<990", "990-1000", "1000-1010", "1010-1020", ">=1020"] + sub["bin"] = pd.cut(sub["pr"], bins=bins, labels=labels, right=False) + out = ( + sub.groupby("bin", observed=True)["dist"] + .agg(["count", "mean", "median"]) + .round(1) + .rename(columns={"count": "n", "mean": "avg_km", "median": "p50_km"}) + .reset_index() + ) + return out + + +def redact(dsn: str) -> str: + if "@" not in dsn: + return dsn + head, tail = dsn.split("@", 1) + if "://" in head and ":" in head.split("://", 1)[1]: + scheme, rest = head.split("://", 1) + user = rest.split(":", 1)[0] + return f"{scheme}://{user}:***@{tail}" + return dsn + + +# ─── Main ─────────────────────────────────────────────────────────────────── + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--dsn", + default=os.environ.get("PROP_PROD_DB_URL"), + help="Postgres connection string (defaults to $PROP_PROD_DB_URL)", + ) + parser.add_argument( + "--out", + default=None, + help="Output Markdown path (defaults to docs/algo-reports/YYYY-MM-DD-recalibration.md)", + ) + parser.add_argument( + "--statement-timeout", + default="20min", + help="Postgres statement_timeout (default: 20min)", + ) + args = parser.parse_args() + + if not args.dsn: + print( + "error: PROP_PROD_DB_URL not set and --dsn not given; populate " + "via .envrc", + file=sys.stderr, + ) + return 2 + + today = dt.date.today().isoformat() + out_path = Path( + args.out + or Path(__file__).resolve().parent.parent + / "docs" + / "algo-reports" + / f"{today}-recalibration.md" + ) + out_path.parent.mkdir(parents=True, exist_ok=True) + + print(f"connecting to {redact(args.dsn)}", file=sys.stderr) + + sections: list[str] = [] + sections.append( + REPORT_HEADER.format( + today=today, + dsn_redacted=redact(args.dsn), + stmt_timeout=args.statement_timeout, + ) + ) + + with psycopg.connect(args.dsn, autocommit=True) as conn: + with conn.cursor() as cur: + cur.execute(f"SET statement_timeout = '{args.statement_timeout}';") + + print("• row counts", file=sys.stderr) + sections.append("\n## Row counts and date ranges\n") + sections.append(md_table(fetch_df(conn, ROW_COUNTS_SQL))) + + print("• data gaps", file=sys.stderr) + gaps = fetch_df(conn, DATA_GAP_SQL).iloc[0].to_dict() + sections.append("\n## Data gaps (these should NOT be zero in a healthy prod)\n") + sections.append( + "\n".join(f"- **{k}**: {v}" for k, v in gaps.items()) + "\n" + ) + + print("• contacts by band", file=sys.stderr) + sections.append("\n## Contacts by band (≥902 MHz)\n") + sections.append(md_table(fetch_df(conn, CONTACTS_BY_BAND_SQL))) + + print("• sounding monthly", file=sys.stderr) + sections.append("\n## Monthly sounding ducting\n") + sections.append(md_table(fetch_df(conn, SOUNDING_MONTHLY_SQL))) + + print("• native HRRR ducts", file=sys.stderr) + sections.append( + "\n## HRRR native-profile duct distribution (best supportable band)\n" + ) + sections.append(md_table(fetch_df(conn, NATIVE_DUCT_SQL))) + + print("• per-band correlations (this is the slow one)", file=sys.stderr) + joined = fetch_df(conn, PER_BAND_JOIN_SQL) + + if joined.empty: + sections.append("\n_No contacts ↔ HRRR matches were found._\n") + else: + sections.append( + f"\n## Contact ↔ HRRR per-band Pearson correlations " + f"(matched n={len(joined)})\n" + ) + sections.append(md_table(correlations_per_band(joined))) + + sections.append("\n## 10 GHz: HPBL bin distance distribution\n") + sections.append(md_table(hpbl_bins(joined))) + + sections.append("\n## 10 GHz: pressure bin distance distribution\n") + sections.append(md_table(pressure_bins(joined))) + + out_path.write_text("\n".join(sections)) + print(f"wrote {out_path}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test/microwaveprop/propagation/band_config_test.exs b/test/microwaveprop/propagation/band_config_test.exs index dbfe65bf..cebc4530 100644 --- a/test/microwaveprop/propagation/band_config_test.exs +++ b/test/microwaveprop/propagation/band_config_test.exs @@ -3,7 +3,27 @@ defmodule Microwaveprop.Propagation.BandConfigTest do alias Microwaveprop.Propagation.BandConfig - @all_freqs [902, 1_296, 2_304, 3_456, 5_760, 10_000, 24_000, 47_000, 68_000, 75_000, 122_000, 134_000, 241_000] + @all_freqs [ + 902, + 1_296, + 2_304, + 3_456, + 5_760, + 10_000, + 24_000, + 47_000, + 68_000, + 75_000, + 122_000, + 134_000, + 142_000, + 145_000, + 241_000, + 288_000, + 322_000, + 403_000, + 411_000 + ] describe "get/1" do test "returns config for 10 GHz" do @@ -95,12 +115,44 @@ defmodule Microwaveprop.Propagation.BandConfigTest do test "returns nil for unknown band" do assert BandConfig.get(999) == nil end + + test "returns config for 142 GHz (between H2O 118 and 183 lines)" do + config = BandConfig.get(142_000) + assert config.freq_mhz == 142_000 + assert config.humidity_effect == :harmful + assert config.rain_alpha < 1.0 + end + + test "returns config for 145 GHz" do + config = BandConfig.get(145_000) + assert config.freq_mhz == 145_000 + assert config.humidity_effect == :harmful + end + + test "returns config for 288 GHz (sub-mm window)" do + config = BandConfig.get(288_000) + assert config.freq_mhz == 288_000 + assert config.humidity_effect == :harmful + assert config.typical_range_km <= 10 + end + + test "returns config for 322 GHz" do + assert %{freq_mhz: 322_000, humidity_effect: :harmful} = BandConfig.get(322_000) + end + + test "returns config for 403 GHz" do + assert %{freq_mhz: 403_000, humidity_effect: :harmful} = BandConfig.get(403_000) + end + + test "returns config for 411 GHz" do + assert %{freq_mhz: 411_000, humidity_effect: :harmful} = BandConfig.get(411_000) + end end describe "all_bands/0" do - test "returns 13 bands" do + test "returns 19 bands (13 original + 142, 145, 288, 322, 403, 411 GHz)" do bands = BandConfig.all_bands() - assert length(bands) == 13 + assert length(bands) == 19 end test "bands are sorted by frequency" do diff --git a/test/microwaveprop/propagation/scorer_test.exs b/test/microwaveprop/propagation/scorer_test.exs index cad88678..b10afa23 100644 --- a/test/microwaveprop/propagation/scorer_test.exs +++ b/test/microwaveprop/propagation/scorer_test.exs @@ -232,29 +232,56 @@ defmodule Microwaveprop.Propagation.ScorerTest do # ── score_refractivity/3 ───────────────────────────────────────── describe "score_refractivity/3" do + # 750 m sits in the [500, 1500) baseline-multiplier band so existing + # threshold assertions don't pick up the HPBL adjustment. + @baseline_bl 750 + test "strong ducting gradient returns high score for beneficial" do # gradient < -500 -> 98 for beneficial - assert Scorer.score_refractivity(-600, 500, @band_10g) == 98 + assert Scorer.score_refractivity(-600, @baseline_bl, @band_10g) == 98 end test "strong gradient returns 85 for harmful" do # gradient < -200 -> 85 for harmful (HRRR-calibrated) - assert Scorer.score_refractivity(-250, 500, @band_24g) == 85 + assert Scorer.score_refractivity(-250, @baseline_bl, @band_24g) == 85 end test "moderate gradient returns appropriate score" do # gradient < -150 but >= -200 -> 92 for beneficial (HRRR-calibrated) - assert Scorer.score_refractivity(-160, 500, @band_10g) == 92 + assert Scorer.score_refractivity(-160, @baseline_bl, @band_10g) == 92 end test "nil gradient returns 50" do - assert Scorer.score_refractivity(nil, 500, @band_10g) == 50 + assert Scorer.score_refractivity(nil, @baseline_bl, @band_10g) == 50 end - test "weak gradient returns default regardless of BL depth" do - # gradient > -40 (no threshold match) -> default 42 - assert Scorer.score_refractivity(-30, 200, @band_10g) == 42 - assert Scorer.score_refractivity(-30, 500, @band_10g) == 42 + test "nil bl_depth_m treated as baseline (no HPBL multiplier)" do + # A nil HPBL must not crash and must produce the unmultiplied score. + assert Scorer.score_refractivity(-160, nil, @band_10g) == 92 + end + + test "shallow boundary layer (<200 m) boosts the gradient score" do + # Same -100 gradient: shallow BL should clearly out-score a baseline BL. + shallow = Scorer.score_refractivity(-100, 150, @band_10g) + baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g) + assert shallow > baseline + assert shallow <= 100 + end + + test "deep boundary layer (>=2000 m) penalises the gradient score" do + # Same -100 gradient: deep BL drags the score down. + deep = Scorer.score_refractivity(-100, 2200, @band_10g) + baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g) + assert deep < baseline + assert deep >= 0 + end + + test "shallow BL also lifts the weak-gradient default" do + # The HPBL multiplier applies to the default fallback too — the binned + # data shows shallow-BL paths run ~230 km even when HRRR gradient is weak. + shallow_default = Scorer.score_refractivity(-30, 150, @band_10g) + baseline_default = Scorer.score_refractivity(-30, @baseline_bl, @band_10g) + assert shallow_default > baseline_default end end diff --git a/test/microwaveprop/propagation_test.exs b/test/microwaveprop/propagation_test.exs index 1875ccac..130ea54d 100644 --- a/test/microwaveprop/propagation_test.exs +++ b/test/microwaveprop/propagation_test.exs @@ -31,7 +31,7 @@ defmodule Microwaveprop.PropagationTest do valid_time = ~U[2026-07-15 13:00:00Z] results = Propagation.score_grid_point(hrrr_profile, valid_time, 33.0, -97.0) - assert length(results) == 13 + assert length(results) == 19 Enum.each(results, fn result -> assert result.score >= 0 and result.score <= 100 diff --git a/test/microwaveprop/workers/era5_month_batch_worker_test.exs b/test/microwaveprop/workers/era5_month_batch_worker_test.exs index a36ba94a..699a86ea 100644 --- a/test/microwaveprop/workers/era5_month_batch_worker_test.exs +++ b/test/microwaveprop/workers/era5_month_batch_worker_test.exs @@ -3,6 +3,12 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorkerTest do alias Microwaveprop.Workers.Era5MonthBatchWorker + describe "worker config" do + test "max_attempts bumped to 5 so CDS slowness doesn't discard tiles after 3 tries" do + assert Era5MonthBatchWorker.__opts__()[:max_attempts] == 5 + end + end + describe "unique constraint" do test "collapses identical (year, month, tile_lat, tile_lon) jobs" do args = %{year: 2014, month: 3, tile_lat: 32, tile_lon: -98}