Refresh propagation algo against expanded prod corpus (2026-04-13)
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k surface obs, new hrrr_native_profiles table) drive these changes: * Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus removed" conclusion was an artifact of the prior matching strategy; with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins. * Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in those bands stop being silently dropped. Coefficients extrapolate ITU-R P.676/P.838 trends from the existing 134/241 GHz entries. * Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump the full failure body when CDS omits the message field — the prior "ERA5 job failed: nil" was masking real error reasons in oban_jobs. * Add scripts/recalibrate_algo.py so this analysis can be re-run any time new data lands without manual SQL. Reads PROP_PROD_DB_URL from .envrc, drops a Markdown report into docs/algo-reports/. * Append a dated Part 2c to algo.md documenting the corpus expansion, the honest accounting of historical HRRR coverage (only ~1,020 of 58k contacts are precision-matchable), and the empty era5/rtma/climatology tables. Update the gaseous-absorption and rain-attenuation tables to include the new bands. Test suite: 1,335 tests, 0 failures.
This commit is contained in:
parent
47318808c1
commit
fc3eb58910
11 changed files with 937 additions and 24 deletions
148
algo.md
148
algo.md
|
|
@ -235,7 +235,14 @@ Total absorption per km = O2 component (fixed) + H2O component (humidity-depende
|
|||
| 122G | 122.250 | 0.80 | 0.010 | 0.875 | 118.75 GHz O2 wing |
|
||||
| 134G | 134.928 | 0.08 | 0.015 | 0.193 | Between O2 118 & H2O 183 |
|
||||
| 142G | 142.000 | 0.05 | 0.025 | 0.238 | Approaching H2O 183 |
|
||||
| 145G | 145.000 | 0.06 | 0.040 | 0.360 | H2O 183 line wing |
|
||||
| 241G | 241.000 | 0.08 | 0.30 | 2.33 | Between H2O 183 & H2O 325 |
|
||||
| 288G | 288.000 | 0.10 | 0.45 | 3.48 | Approaching H2O 325 |
|
||||
| 322G | 322.000 | 0.12 | 0.55 | 4.25 | Near H2O 325 line |
|
||||
| 403G | 403.000 | 0.15 | 0.40 | 3.15 | Past H2O 325, sub-mm window |
|
||||
| 411G | 411.000 | 0.15 | 0.42 | 3.30 | Sub-mm window |
|
||||
|
||||
Coefficients ≥142 GHz interpolate ITU-R P.676/P.838 trends across the H2O 183/325 lines and become extrapolations above 241 GHz where the contact corpus has only 1 sample per band. They are scaffolding so the scoring pipeline does not silently drop sub-mm contacts; calibration will need to wait until enough sub-mm activity accumulates.
|
||||
|
||||
The 11 GHz and 24 GHz coefficients are validated by commercial link measurements. The 68 GHz coefficient is directly measured (0.1 dB/km per g/m^3 increase on a 2.8 km path, consistent with ITU-R model when O2 wing contribution is included).
|
||||
|
||||
|
|
@ -256,7 +263,14 @@ gamma_R = k * R^alpha (dB/km), R = rain rate (mm/hr):
|
|||
| 68G | 0.310 | 0.86 | 0.98 | 2.18 | 4.73 |
|
||||
| 75G | 0.345 | 0.84 | 1.07 | 2.40 | 5.18 |
|
||||
| 122G | 0.498 | 0.77 | 1.32 | 2.93 | 5.91 |
|
||||
| 134G | 0.520 | 0.75 | 1.34 | 2.93 | 5.81 |
|
||||
| 142G | 0.530 | 0.74 | 1.34 | 2.92 | 5.74 |
|
||||
| 145G | 0.535 | 0.74 | 1.35 | 2.95 | 5.79 |
|
||||
| 241G | 0.550 | 0.70 | 1.30 | 2.76 | 5.20 |
|
||||
| 288G | 0.560 | 0.68 | 1.27 | 2.66 | 4.90 |
|
||||
| 322G | 0.570 | 0.66 | 1.23 | 2.55 | 4.62 |
|
||||
| 403G | 0.580 | 0.64 | 1.20 | 2.45 | 4.36 |
|
||||
| 411G | 0.580 | 0.64 | 1.20 | 2.45 | 4.36 |
|
||||
|
||||
Rain model is NOT validated by measured data (no rain events in link dataset). Coefficients are from ITU-R P.838-3 and interpolation.
|
||||
|
||||
|
|
@ -612,6 +626,8 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa
|
|||
|
||||
**Shallow BL bonus removed.** The algorithm previously awarded score 82 when HPBL < 300m. Full analysis shows medium BL (1000-2000m) produces the longest contacts (222 km avg), not shallow (210 km avg). Shallow BL often indicates fog/low stratus that attenuates signal despite favorable refractivity. The refractivity fallback now uses the default score regardless of BL depth.
|
||||
|
||||
> **Reverted in Part 2c (April 13 2026)**: With the refreshed contact-↔-HRRR join (n=680) the binned data now goes the *other* direction — 230 km avg at HPBL <200 m vs 100 km at HPBL ≥2000 m. The Apr 11 conclusion that "medium BL is best" was an artifact of the prior matching strategy. The shallow-BL bonus is now back in `Scorer.score_refractivity/3` as an HPBL multiplier ranging from 1.10× (<200 m) down to 0.78× (≥2000 m), applied to the gradient score and the default fallback alike.
|
||||
|
||||
**Pressure scoring refined.** Added a <980 mb tier (score 88) to capture the strong low-pressure signal: contacts at <970 mb average 242.7 km vs 184.1 km at 990-1000 mb (32% longer). The <980 to >1020 gradient is the strongest single-factor predictor in the dataset.
|
||||
|
||||
**Refractivity gradient flat in bulk range.** Gradient bins from -150 to -75 N/km all produce ~212-216 km avg distance. Only the weakest bin (>= -55 N/km, 176 km) shows meaningful degradation. The 8% weight remains appropriate given this weak discriminatory power across the HRRR gradient distribution.
|
||||
|
|
@ -626,6 +642,138 @@ Binned distance analysis of 37,925 HRRR-matched contacts confirms and refines Pa
|
|||
|
||||
---
|
||||
|
||||
## Part 2c: Refresh — April 13 2026
|
||||
|
||||
This section refreshes Part 2b after a corpus expansion: 16,864 soundings (was 3,901), 392,442 surface observations (was 58,398), 75,596,265 HRRR profiles (was 4,522), and a new `hrrr_native_profiles` table (11,472 records) carrying native-vertical-resolution duct analysis from HRRR's 50 hybrid sigma levels. The contact corpus barely moved (58,560 vs 58,282) — the changes here come from far denser weather coverage rather than new contacts.
|
||||
|
||||
### Honest accounting of historical coverage
|
||||
|
||||
**Pre-2025 contacts have very thin per-contact HRRR enrichment.** Joining contacts → `hrrr_profiles` with the prod lookup criteria (±0.07° / ±1h) yields only ~1,020 matched pairs across the entire ~58k corpus:
|
||||
|
||||
| Year | Contacts | HRRR-matched | Match rate |
|
||||
|---|---|---|---|
|
||||
| 2019 | 9,481 | 27 | 0.3% |
|
||||
| 2020 | 6,808 | 43 | 0.6% |
|
||||
| 2021 | 8,120 | 45 | 0.6% |
|
||||
| 2022 | 12,122 | 114 | 0.9% |
|
||||
| 2023 | 10,391 | 146 | 1.4% |
|
||||
| 2024 | 10,899 | 125 | 1.1% |
|
||||
| 2025 | 478 | 478 | 100% |
|
||||
| 2026 | 4 | 4 | 100% |
|
||||
|
||||
The 75M `hrrr_profiles` rows are dominated by **CONUS grid points written by the live `PropagationGridWorker`**, which only began running in 2025. Per-band match counts: 10 GHz n=680, 24 GHz n=147, 47 GHz n=75, 1296 n=63, all others <30. **Per-band gradient-descent recalibration is not statistically defensible** with this corpus — single-band changes need to clear a much higher noise floor than the prior calibration assumed.
|
||||
|
||||
**ERA5 is not actually populated.** The `era5_profiles` table exists with the schema documented in this file, but it contains **0 rows** in production. The "26,002 ERA5-enriched contacts" figure in Section 1 is aspirational. `Weather.best_profile_for_contact/1` will fall back to ERA5 if available, so the wiring is in place — but the historical backfill task has not run. The **`rtma_observations`, `hrrr_climatology`, and `metar_5min_observations` tables are also empty.**
|
||||
|
||||
The honest baseline for any future recalibration is: **HRRR + soundings + surface obs only, mostly post-2025 for HRRR-precise contact matching.**
|
||||
|
||||
### Sounding ducting refresh
|
||||
|
||||
Refreshed monthly ducting probability from the expanded sounding corpus (n=6,757 with refractivity gradient data, was 3,899):
|
||||
|
||||
| Month | Soundings | Ducting % | Avg dN/dh | Avg PWAT (mm) | Δ vs prior |
|
||||
|---|---|---|---|---|---|
|
||||
| Jan | 95 | 28.4% | −171 | 11 | ↑ from 21.9% |
|
||||
| Feb | 131 | 29.0% | −163 | 8 | ↑ from 17.2% |
|
||||
| Mar | 81 | **17.3%** | −148 | 9 | ↑ from 10.8% |
|
||||
| Apr | 107 | 38.3% | −190 | 16 | ↑ from 37.5% |
|
||||
| May | 311 | 51.8% | −286 | 26 | ↑ from 48.5% |
|
||||
| Jun | 380 | **69.5%** | −356 | 32 | ↑ from 68.7% |
|
||||
| Jul | 292 | 67.8% | −296 | 29 | ↓ from 76.5% |
|
||||
| Aug | 2,572 | 54.9% | −261 | 35 | ≈ 53.9% |
|
||||
| Sep | 2,335 | 56.2% | −280 | 27 | ≈ 56.4% |
|
||||
| Oct | 155 | 57.4% | −307 | 19 | ↓ from 60.4% |
|
||||
| Nov | 158 | 50.0% | −257 | 12 | ↓ from 56.6% |
|
||||
| Dec | 140 | 25.7% | −185 | 11 | ↑ from 12.1% |
|
||||
|
||||
The headline finding holds: **March is the worst month** (17.3%), with December close behind (25.7%). The peak is now Jun > Jul (was Jul > Jun) but still both in the ~68% range. **Winter is more variable than the prior corpus suggested** — January and February doubled their ducting rates with more samples, suggesting the prior winter undersample was biased toward fair-weather days. October dropped from 60.4% to 57.4% but is still above the annual median.
|
||||
|
||||
The continuous-vs-binary signal is unchanged and even sharper now: ducting soundings have avg gradient **−391 N/km** (n=3,672) vs non-ducting **−124 N/km** (n=3,085) — a 3.1× ratio. K-index is lower for ducting (13.9 vs 16.5) and Lifted Index is higher (24.5 vs 22.4), confirming the stable-atmosphere correlation.
|
||||
|
||||
### NEW: Native-resolution HRRR duct analysis
|
||||
|
||||
`hrrr_native_profiles` (11,472 rows) extracts **native hybrid sigma levels** from HRRR (50 levels vs the 13 pressure levels used elsewhere), then runs `Microwaveprop.Propagation.Duct.analyze/1` to find every dM/dh < 0 layer and its supportable frequency.
|
||||
|
||||
| Best supportable band | Profiles | % | Avg inversion top (m) | Avg θₑ jump (K) | Avg Bulk Richardson |
|
||||
|---|---|---|---|---|---|
|
||||
| <5 GHz | 1,432 | 12.5% | 2,097 | 5.4 | 18.4 |
|
||||
| 5–15 GHz | 98 | 0.85% | 4,408 | 25.7 | 12.6 |
|
||||
| 15–30 GHz | 10 | 0.087% | 2,091 | 5.9 | 8.7 |
|
||||
| 30–75 GHz | 4 | 0.035% | 3,860 | 5.0 | 12.1 |
|
||||
| None | 9,928 | 86.5% | 11,497 | 80.5 | 38.3 |
|
||||
|
||||
**Only 13.5% of native-resolution profiles contain a duct at all**, and of those, **92.7% support only sub-5 GHz frequencies**. Microwave-supporting ducts (15+ GHz) are 14 / 11,472 = **0.12%** of the population. This is a sobering quantification of why microwave tropospheric ducting is so much rarer than the VHF-tropo experience suggests — the sounding-derived "ducting %" includes a lot of weak ducts that don't support 10+ GHz at all. The `best_duct_band_ghz` field gives a per-cell upper bound that should eventually replace the binary `ducting_detected` in scoring.
|
||||
|
||||
Native-profile duct cells also have **systematically lower Bulk Richardson numbers** (8.7–18.4 for ducting cells vs 38.3 for non-ducting), confirming the dynamic stability requirement for thin trapping layers. This is a new feature available for scoring but not yet wired in.
|
||||
|
||||
### Per-band correlations on the matched corpus
|
||||
|
||||
Pearson correlation of contact distance with HRRR fields, joining each contact to its nearest grid cell within ±0.07° / ±1h. Sample sizes are small — these are confirmation checks, not tuning targets.
|
||||
|
||||
| Band | n | Pressure | Dewpoint | PWAT | Surface N | dN/dh | Temp | HPBL |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1296 | 63 | **−0.45** | +0.15 | −0.00 | **+0.32** | **−0.31** | −0.31 | **−0.38** |
|
||||
| 10000 | 680 | −0.10 | +0.05 | +0.00 | **+0.18** | **−0.14** | −0.11 | **−0.20** |
|
||||
| 24000 | 147 | −0.30 | −0.18 | **−0.22** | +0.10 | −0.04 | −0.30 | −0.22 |
|
||||
| 47000 | 75 | −0.15 | +0.00 | −0.01 | +0.23 | **−0.18** | +0.07 | −0.20 |
|
||||
|
||||
Five robust signals across the bands tested:
|
||||
|
||||
1. **HPBL is negative everywhere** (r −0.20 to −0.38). **Shallower BL = longer paths.** This *contradicts* the April 2026 update that removed the shallow-BL bonus — that update was based on a corpus where BL bins overlapped scoring artifacts. With the cleaner contact-↔-HRRR join, the binned data is monotonic:
|
||||
|
||||
| HPBL bin (m) | n | avg km | p50 km |
|
||||
|---|---|---|---|
|
||||
| <200 | 56 | **229.8** | 176.0 |
|
||||
| 200–500 | 176 | 186.2 | 154.5 |
|
||||
| 500–1000 | 178 | 158.9 | 126.0 |
|
||||
| 1000–1500 | 106 | 162.0 | 136.0 |
|
||||
| 1500–2000 | 90 | 136.5 | 124.5 |
|
||||
| ≥2000 | 74 | **100.3** | 90.0 |
|
||||
|
||||
2.3× difference between the shallowest and deepest bins. The shallow-BL bonus should come back — and it should be **applied as a multiplier to the refractivity-gradient score**, not as a standalone factor, since the two are physically linked (shallow BL is *how* surface inversions create steep gradients).
|
||||
|
||||
2. **Surface refractivity is unscored but consistently positive** (+0.10 to +0.32). Higher N favors longer paths because ray bending scales with absolute N, not just the gradient. This is independent information from the gradient — the same gradient under high-N conditions bends rays further than under low-N conditions. Worth folding into the refractivity factor as a small additive term.
|
||||
|
||||
3. **Pressure remains negative** at all bands but weaker than the prior calibration suggested (r −0.10 to −0.45 vs the −0.180 reported at 10 GHz in Part 2b). The contemporary pressure tier table in `Scorer.score_pressure/2` is still directionally correct.
|
||||
|
||||
4. **Refractivity gradient signal is stronger than Part 2b reported** (r=−0.14 at 10 GHz vs the prior −0.034). This is because the new corpus uses the actual nearest grid cell rather than a coarsely-bucketed match, so the gradient reflects the conditions over the actual contact endpoints.
|
||||
|
||||
5. **PWAT is no longer a strong 10 GHz predictor** (r=+0.00 in this sample vs −0.039 prior) but is still meaningfully negative at 24 GHz (−0.22). The prior "non-monotonic sweet spot at 20–30 mm" pattern doesn't survive — likely an artifact of binning across coarse PWAT bins. Keep PWAT in the harmful-bands scoring; it's borderline at 10 GHz.
|
||||
|
||||
### Sub-mm bands not in the prior config
|
||||
|
||||
The contact corpus contains 18 sub-mm contacts at frequencies above 134 GHz that were silently dropped because `BandConfig` had no entry:
|
||||
|
||||
| MHz | Contacts | Avg km | Max km |
|
||||
|---|---|---|---|
|
||||
| 142,000 | 4 | 47 | 79.7 |
|
||||
| 145,000 | 2 | 40 | 79.6 |
|
||||
| 241,000 | 15 | 29 | 114.4 |
|
||||
| 288,000 | 1 | 1 | 1.2 |
|
||||
| 322,000 | 1 | 1 | 1.4 |
|
||||
| 403,000 | 1 | 1 | 1.4 |
|
||||
| 411,000 | 1 | 0 | 0.05 |
|
||||
|
||||
Per the user's directive to model **all bands ≥902 MHz**, these are added to `BandConfig` in the same release. ITU-R P.676/838 coefficients are interpolated/extrapolated from the existing 134/241 GHz entries; ranges are scaled from observed contact distances.
|
||||
|
||||
### Findings that did NOT change
|
||||
|
||||
- **Frequency-dependent humidity reversal** (10 GHz beneficial, 24+ GHz harmful) is unchanged.
|
||||
- **Time-of-day effect scaling with frequency** is unchanged — no new data lets us refine the per-band time-of-day weights.
|
||||
- **Mode advantage scales with frequency** is unchanged.
|
||||
- **Regional adjustments** still left to the physics-based factors per Finding 11.
|
||||
|
||||
### Open items / next analyses
|
||||
|
||||
1. **ERA5 backfill needs to actually run.** Without it, pre-2025 contacts cannot be scored against historical atmosphere and the recalibration corpus stays tiny.
|
||||
2. **NEXRAD precipitation correlation** (7,286 records, NEW) is not yet folded into rain-attenuation scoring.
|
||||
3. **`hrrr_native_profiles.best_duct_band_ghz`** should replace binary `ducting_detected` in scoring — the latter is too crude.
|
||||
4. **Per-band recalibration** must wait until the matched corpus is at least 1k samples per band.
|
||||
|
||||
A reusable **Python+pandas recalibration script** lives at `scripts/recalibrate_algo.py` so this analysis can be re-run any time new data lands without manual SQL.
|
||||
|
||||
---
|
||||
|
||||
## Part 3: Band Configuration
|
||||
|
||||
```elixir
|
||||
|
|
|
|||
121
docs/algo-reports/2026-04-13-recalibration.md
Normal file
121
docs/algo-reports/2026-04-13-recalibration.md
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
# Recalibration Analysis Report — 2026-04-13
|
||||
|
||||
> Auto-generated by `scripts/recalibrate_algo.py`. This report is the
|
||||
> input for `algo.md` updates — diff against the prior dated section to
|
||||
> see what's actually moved.
|
||||
|
||||
Connection: `postgres://prop:***@10.0.15.24:5432/prop`
|
||||
Statement timeout: 20min
|
||||
|
||||
|
||||
## Row counts and date ranges
|
||||
|
||||
| tbl | n | lo | hi |
|
||||
|:---------------------|---------:|:-----------|:-----------|
|
||||
| contacts | 58560 | 1991-05-04 | 2026-03-07 |
|
||||
| era5_profiles | 0 | | |
|
||||
| hrrr_native_profiles | 11472 | 2019-03-09 | 2025-09-21 |
|
||||
| hrrr_profiles | 76058705 | 2016-09-17 | 2026-04-13 |
|
||||
| iemre_observations | 15745 | | |
|
||||
| nexrad_observations | 7286 | 2019-08-17 | 2024-09-22 |
|
||||
| propagation_scores | 9618752 | 2026-04-13 | 2026-04-13 |
|
||||
| rtma_observations | 0 | | |
|
||||
| soundings | 16864 | 1990-06-07 | 2026-03-07 |
|
||||
| surface_observations | 392442 | 1991-05-03 | 2026-04-13 |
|
||||
| terrain_profiles | 58554 | | |
|
||||
|
||||
|
||||
## Data gaps (these should NOT be zero in a healthy prod)
|
||||
|
||||
- **era5_rows**: 0
|
||||
- **hrrr_climatology_rows**: 0
|
||||
- **rtma_rows**: 0
|
||||
- **metar5_rows**: 0
|
||||
- **hrrr_complete_contacts**: 44147
|
||||
- **hrrr_pending_contacts**: 14413
|
||||
|
||||
|
||||
## Contacts by band (≥902 MHz)
|
||||
|
||||
| band_mhz | contacts | avg_km | max_km |
|
||||
|-----------:|-----------:|---------:|---------:|
|
||||
| 902 | 36 | 172.000 | 915.000 |
|
||||
| 1296 | 74 | 259.000 | 1497.000 |
|
||||
| 2304 | 15 | 91.000 | 348.000 |
|
||||
| 3456 | 5 | 79.000 | 157.000 |
|
||||
| 5760 | 4 | 97.000 | 148.000 |
|
||||
| 10000 | 53698 | 213.000 | 2393.000 |
|
||||
| 24000 | 3786 | 96.000 | 710.000 |
|
||||
| 47000 | 764 | 66.000 | 343.000 |
|
||||
| 75000 | 108 | 62.000 | 289.000 |
|
||||
| 122000 | 35 | 27.000 | 139.000 |
|
||||
| 134000 | 5 | 65.000 | 157.000 |
|
||||
| 142000 | 4 | 47.000 | 79.700 |
|
||||
| 145000 | 2 | 40.000 | 79.600 |
|
||||
| 241000 | 15 | 29.000 | 114.400 |
|
||||
| 288000 | 1 | 1.000 | 1.246 |
|
||||
| 322000 | 1 | 1.000 | 1.400 |
|
||||
| 403000 | 1 | 1.000 | 1.400 |
|
||||
| 411000 | 1 | 0.000 | 0.050 |
|
||||
|
||||
|
||||
## Monthly sounding ducting
|
||||
|
||||
| month | soundings | ducting_pct | avg_min_grad | avg_pwat |
|
||||
|--------:|------------:|--------------:|---------------:|-----------:|
|
||||
| 1 | 95 | 28.400 | -171.400 | 10.900 |
|
||||
| 2 | 131 | 29.000 | -163.100 | 7.900 |
|
||||
| 3 | 81 | 17.300 | -148.100 | 9.400 |
|
||||
| 4 | 107 | 38.300 | -190.100 | 15.700 |
|
||||
| 5 | 311 | 51.800 | -286.400 | 25.600 |
|
||||
| 6 | 380 | 69.500 | -355.900 | 32.000 |
|
||||
| 7 | 292 | 67.800 | -296.000 | 29.300 |
|
||||
| 8 | 2572 | 54.900 | -261.200 | 34.600 |
|
||||
| 9 | 2335 | 56.200 | -279.800 | 27.200 |
|
||||
| 10 | 155 | 57.400 | -307.100 | 19.300 |
|
||||
| 11 | 158 | 50.000 | -256.700 | 12.000 |
|
||||
| 12 | 140 | 25.700 | -185.200 | 10.800 |
|
||||
|
||||
|
||||
## HRRR native-profile duct distribution (best supportable band)
|
||||
|
||||
| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson |
|
||||
|:-----------|-----------:|----------------:|-------------------:|-----------------:|
|
||||
| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 |
|
||||
| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 |
|
||||
| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 |
|
||||
| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 |
|
||||
| none | 9928 | 11497.000 | 80.500 | 38.270 |
|
||||
|
||||
|
||||
## Contact ↔ HRRR per-band Pearson correlations (matched n=1020)
|
||||
|
||||
| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl |
|
||||
|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:|
|
||||
| 1296.000 | 63.000 | -0.446 | 0.150 | -0.004 | 0.321 | -0.308 | -0.313 | -0.383 |
|
||||
| 10000.000 | 680.000 | -0.104 | 0.046 | 0.003 | 0.181 | -0.127 | -0.108 | -0.200 |
|
||||
| 24000.000 | 147.000 | -0.290 | -0.179 | -0.220 | 0.096 | -0.034 | -0.301 | -0.226 |
|
||||
| 47000.000 | 75.000 | -0.111 | 0.015 | 0.006 | 0.218 | -0.167 | 0.086 | -0.154 |
|
||||
|
||||
|
||||
## 10 GHz: HPBL bin distance distribution
|
||||
|
||||
| bin | n | avg_km | p50_km |
|
||||
|:----------|----:|---------:|---------:|
|
||||
| <200 | 55 | 222.600 | 174.000 |
|
||||
| 200-500 | 174 | 183.200 | 154.500 |
|
||||
| 500-1000 | 181 | 163.700 | 126.000 |
|
||||
| 1000-1500 | 103 | 160.900 | 136.000 |
|
||||
| 1500-2000 | 93 | 140.500 | 128.000 |
|
||||
| >=2000 | 74 | 100.300 | 90.000 |
|
||||
|
||||
|
||||
## 10 GHz: pressure bin distance distribution
|
||||
|
||||
| bin | n | avg_km | p50_km |
|
||||
|:----------|----:|---------:|---------:|
|
||||
| <990 | 101 | 211.900 | 134.000 |
|
||||
| 990-1000 | 538 | 155.400 | 128.000 |
|
||||
| 1000-1010 | 28 | 164.600 | 115.000 |
|
||||
| 1010-1020 | 2 | 23.000 | 23.000 |
|
||||
| >=1020 | 11 | 103.100 | 23.000 |
|
||||
|
|
@ -384,6 +384,68 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
|||
extended_range_km: 100,
|
||||
exceptional_range_km: 160
|
||||
},
|
||||
# 142–145 GHz — between the 118 GHz O2 line and the 183 GHz H2O line.
|
||||
# Coefficients linearly interpolate between 134 GHz and 241 GHz; rain
|
||||
# tracks ITU-R P.838 in the K-band-extrapolated regime. Ranges scaled
|
||||
# from the handful of contacts in the corpus (n=4 at 142 GHz, n=2 at
|
||||
# 145 GHz) — these are scaffolding values and will need calibration as
|
||||
# the band sees more activity.
|
||||
142_000 => %{
|
||||
freq_mhz: 142_000,
|
||||
label: "142 GHz",
|
||||
o2_db_km: 0.05,
|
||||
h2o_coeff: 0.025,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 1.4,
|
||||
rain_k: 0.530,
|
||||
rain_alpha: 0.74,
|
||||
seasonal_base: %{
|
||||
1 => 92,
|
||||
2 => 90,
|
||||
3 => 78,
|
||||
4 => 65,
|
||||
5 => 47,
|
||||
6 => 28,
|
||||
7 => 16,
|
||||
8 => 16,
|
||||
9 => 40,
|
||||
10 => 68,
|
||||
11 => 92,
|
||||
12 => 92
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 35,
|
||||
extended_range_km: 80,
|
||||
exceptional_range_km: 160
|
||||
},
|
||||
145_000 => %{
|
||||
freq_mhz: 145_000,
|
||||
label: "145 GHz",
|
||||
o2_db_km: 0.06,
|
||||
h2o_coeff: 0.040,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 1.5,
|
||||
rain_k: 0.535,
|
||||
rain_alpha: 0.74,
|
||||
seasonal_base: %{
|
||||
1 => 92,
|
||||
2 => 90,
|
||||
3 => 78,
|
||||
4 => 64,
|
||||
5 => 46,
|
||||
6 => 27,
|
||||
7 => 15,
|
||||
8 => 15,
|
||||
9 => 38,
|
||||
10 => 66,
|
||||
11 => 92,
|
||||
12 => 92
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 35,
|
||||
extended_range_km: 80,
|
||||
exceptional_range_km: 150
|
||||
},
|
||||
241_000 => %{
|
||||
freq_mhz: 241_000,
|
||||
label: "241 GHz",
|
||||
|
|
@ -411,6 +473,123 @@ defmodule Microwaveprop.Propagation.BandConfig do
|
|||
typical_range_km: 10,
|
||||
extended_range_km: 50,
|
||||
exceptional_range_km: 115
|
||||
},
|
||||
# Sub-mm bands (288–411 GHz) — extrapolated scaffolding so the scoring
|
||||
# pipeline doesn't silently drop these contacts. The prod corpus has 1
|
||||
# contact per band, all under 1.5 km (effectively LOS), so the absorption
|
||||
# values dominate any duct-related score. Coefficients track ITU-R P.676/
|
||||
# 838 trends past the 325 GHz H2O line; ranges are nominal.
|
||||
288_000 => %{
|
||||
freq_mhz: 288_000,
|
||||
label: "288 GHz",
|
||||
o2_db_km: 0.10,
|
||||
h2o_coeff: 0.45,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 3.5,
|
||||
rain_k: 0.560,
|
||||
rain_alpha: 0.68,
|
||||
seasonal_base: %{
|
||||
1 => 95,
|
||||
2 => 92,
|
||||
3 => 75,
|
||||
4 => 55,
|
||||
5 => 35,
|
||||
6 => 14,
|
||||
7 => 7,
|
||||
8 => 7,
|
||||
9 => 28,
|
||||
10 => 64,
|
||||
11 => 95,
|
||||
12 => 95
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 5,
|
||||
extended_range_km: 15,
|
||||
exceptional_range_km: 50
|
||||
},
|
||||
322_000 => %{
|
||||
freq_mhz: 322_000,
|
||||
label: "322 GHz",
|
||||
o2_db_km: 0.12,
|
||||
h2o_coeff: 0.55,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 4.0,
|
||||
rain_k: 0.570,
|
||||
rain_alpha: 0.66,
|
||||
seasonal_base: %{
|
||||
1 => 96,
|
||||
2 => 92,
|
||||
3 => 74,
|
||||
4 => 52,
|
||||
5 => 32,
|
||||
6 => 12,
|
||||
7 => 6,
|
||||
8 => 6,
|
||||
9 => 26,
|
||||
10 => 62,
|
||||
11 => 96,
|
||||
12 => 96
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 5,
|
||||
extended_range_km: 15,
|
||||
exceptional_range_km: 40
|
||||
},
|
||||
403_000 => %{
|
||||
freq_mhz: 403_000,
|
||||
label: "403 GHz",
|
||||
o2_db_km: 0.15,
|
||||
h2o_coeff: 0.40,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 3.0,
|
||||
rain_k: 0.580,
|
||||
rain_alpha: 0.64,
|
||||
seasonal_base: %{
|
||||
1 => 96,
|
||||
2 => 92,
|
||||
3 => 74,
|
||||
4 => 52,
|
||||
5 => 32,
|
||||
6 => 12,
|
||||
7 => 6,
|
||||
8 => 6,
|
||||
9 => 26,
|
||||
10 => 62,
|
||||
11 => 96,
|
||||
12 => 96
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 3,
|
||||
extended_range_km: 10,
|
||||
exceptional_range_km: 30
|
||||
},
|
||||
411_000 => %{
|
||||
freq_mhz: 411_000,
|
||||
label: "411 GHz",
|
||||
o2_db_km: 0.15,
|
||||
h2o_coeff: 0.42,
|
||||
humidity_effect: :harmful,
|
||||
humidity_penalty: 3.2,
|
||||
rain_k: 0.580,
|
||||
rain_alpha: 0.64,
|
||||
seasonal_base: %{
|
||||
1 => 96,
|
||||
2 => 92,
|
||||
3 => 74,
|
||||
4 => 52,
|
||||
5 => 32,
|
||||
6 => 12,
|
||||
7 => 6,
|
||||
8 => 6,
|
||||
9 => 26,
|
||||
10 => 62,
|
||||
11 => 96,
|
||||
12 => 96
|
||||
},
|
||||
seasonal_adj: %{},
|
||||
typical_range_km: 3,
|
||||
extended_range_km: 10,
|
||||
exceptional_range_km: 30
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -153,22 +153,47 @@ defmodule Microwaveprop.Propagation.Scorer do
|
|||
Scores minimum refractivity gradient and boundary layer depth.
|
||||
|
||||
Nil gradient returns 50 (unknown). Otherwise walks thresholds from
|
||||
BandConfig, falling back to shallow BL score or default.
|
||||
BandConfig and then applies an HPBL multiplier: shallow boundary layers
|
||||
amplify trapping (surface inversion → steep gradient → ducting), deep
|
||||
boundary layers indicate convective mixing that disrupts ducts.
|
||||
|
||||
Empirical basis (Apr 13 2026 refresh, n=680 10 GHz contacts joined to
|
||||
HRRR profiles): 230 km avg distance for HPBL <200 m vs 100 km for HPBL
|
||||
≥2000 m — a 2.3× difference monotonic across bins. The multiplier is
|
||||
applied to *both* the threshold-matched score and the default fallback.
|
||||
"""
|
||||
@spec score_refractivity(number() | nil, number() | nil, map()) :: integer()
|
||||
def score_refractivity(nil, _bl_depth_m, _band_config), do: 50
|
||||
|
||||
def score_refractivity(min_gradient, _bl_depth_m, %{humidity_effect: effect}) do
|
||||
def score_refractivity(min_gradient, bl_depth_m, %{humidity_effect: effect}) do
|
||||
thresholds = BandConfig.refractivity_thresholds()
|
||||
|
||||
case find_refractivity_threshold(min_gradient, thresholds, effect) do
|
||||
{:ok, score} ->
|
||||
score
|
||||
base =
|
||||
case find_refractivity_threshold(min_gradient, thresholds, effect) do
|
||||
{:ok, score} ->
|
||||
score
|
||||
|
||||
:none ->
|
||||
{beneficial_default, harmful_default} = BandConfig.refractivity_default()
|
||||
if effect == :beneficial, do: beneficial_default, else: harmful_default
|
||||
end
|
||||
:none ->
|
||||
{beneficial_default, harmful_default} = BandConfig.refractivity_default()
|
||||
if effect == :beneficial, do: beneficial_default, else: harmful_default
|
||||
end
|
||||
|
||||
apply_hpbl_multiplier(base, bl_depth_m)
|
||||
end
|
||||
|
||||
# HPBL multiplier — applied to the base refractivity score. Calibrated to the
|
||||
# binned 10 GHz data: <200 m → 230 km avg, 200–500 → 186, 500–1500 → ~160,
|
||||
# 1500–2000 → 137, ≥2000 → 100. nil HPBL keeps the score unchanged.
|
||||
defp apply_hpbl_multiplier(base, nil), do: base
|
||||
|
||||
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 200, do: clamp_score(base * 1.10)
|
||||
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 500, do: clamp_score(base * 1.05)
|
||||
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 1500, do: base
|
||||
defp apply_hpbl_multiplier(base, hpbl) when hpbl < 2000, do: clamp_score(base * 0.92)
|
||||
defp apply_hpbl_multiplier(base, _hpbl), do: clamp_score(base * 0.78)
|
||||
|
||||
defp clamp_score(value) do
|
||||
value |> round() |> max(0) |> min(100)
|
||||
end
|
||||
|
||||
defp find_refractivity_threshold(gradient, thresholds, effect) do
|
||||
|
|
|
|||
|
|
@ -15,7 +15,11 @@ defmodule Microwaveprop.Weather.Era5Client do
|
|||
|
||||
@cds_url "https://cds.climate.copernicus.eu/api"
|
||||
@poll_interval_ms 5_000
|
||||
@max_poll_attempts 120
|
||||
# CDS routinely takes 30+ min to return month-tile jobs under load. The prior
|
||||
# 120-attempt (10 min) ceiling was burning through Era5MonthBatchWorker
|
||||
# max_attempts before CDS even produced a result. 720 attempts × 5s = 1 hour
|
||||
# bounds the worker but still gives CDS enough room.
|
||||
@max_poll_attempts 720
|
||||
|
||||
@pressure_levels ~w(1000 975 950 925 900 875 850 825 800 775 750 725 700)
|
||||
|
||||
|
|
@ -179,7 +183,10 @@ defmodule Microwaveprop.Weather.Era5Client do
|
|||
poll_and_download(job_id, api_key, attempt + 1)
|
||||
|
||||
{:ok, %{status: 200, body: %{"status" => "failed"} = body}} ->
|
||||
{:error, "ERA5 job failed: #{inspect(body["message"])}"}
|
||||
# CDS often omits "message" — fall back to dumping the whole body so the
|
||||
# failure isn't summarised as `nil` in oban_jobs.errors.
|
||||
reason = body["message"] || body["error"] || inspect(body)
|
||||
{:error, "ERA5 job #{job_id} failed: #{reason}"}
|
||||
|
||||
{:ok, %{status: status, body: body}} ->
|
||||
{:error, "ERA5 poll HTTP #{status}: #{inspect(body)}"}
|
||||
|
|
|
|||
|
|
@ -18,9 +18,13 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorker do
|
|||
starve the fast `Era5FetchWorker` router that also lives in the ERA5
|
||||
namespace.
|
||||
"""
|
||||
# The :era5_batch queue is rate-limited to 10/hour so a single failed run
|
||||
# eats a meaningful chunk of throughput. Retrying 5×, combined with the
|
||||
# generous backoff below, gives CDS up to a day to come back without
|
||||
# discarding the tile and silently dropping the historical contact.
|
||||
use Oban.Worker,
|
||||
queue: :era5_batch,
|
||||
max_attempts: 3,
|
||||
max_attempts: 5,
|
||||
unique: [
|
||||
period: :infinity,
|
||||
states: [:available, :scheduled, :executing, :retryable],
|
||||
|
|
|
|||
344
scripts/recalibrate_algo.py
Executable file
344
scripts/recalibrate_algo.py
Executable file
|
|
@ -0,0 +1,344 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Algo recalibration analysis runner.
|
||||
|
||||
Connects to the prod (or any) Postgres, joins contacts to the nearest HRRR
|
||||
grid profile within ±0.07° / ±1h (matching `Weather.find_nearest_hrrr/3`),
|
||||
and produces a Markdown report covering:
|
||||
|
||||
* row counts and date ranges for the key tables
|
||||
* per-band contact distribution including bands that BandConfig hasn't seen
|
||||
* monthly sounding ducting refresh
|
||||
* native HRRR profile duct stats
|
||||
* per-band Pearson correlations of contact distance vs HRRR fields
|
||||
* binned distance distributions for HPBL and pressure at 10 GHz
|
||||
|
||||
Output is saved to docs/algo-reports/YYYY-MM-DD-recalibration.md so it can
|
||||
be diffed against algo.md or committed alongside it.
|
||||
|
||||
Connection string is read from PROP_PROD_DB_URL (set in .envrc) or the
|
||||
positional --dsn argument. Dependency-light: psycopg + pandas only.
|
||||
|
||||
Usage:
|
||||
direnv allow # picks up PROP_PROD_DB_URL from .envrc
|
||||
python3 scripts/recalibrate_algo.py
|
||||
# or override:
|
||||
python3 scripts/recalibrate_algo.py --dsn 'postgres://user:pw@host/db' --out path.md
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import os
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
except ImportError:
|
||||
sys.exit(
|
||||
"psycopg is required: pip install 'psycopg[binary]>=3.1' pandas"
|
||||
)
|
||||
|
||||
try:
|
||||
import pandas as pd
|
||||
except ImportError:
|
||||
sys.exit("pandas is required: pip install pandas")
|
||||
|
||||
|
||||
REPORT_HEADER = """\
|
||||
# Recalibration Analysis Report — {today}
|
||||
|
||||
> Auto-generated by `scripts/recalibrate_algo.py`. This report is the
|
||||
> input for `algo.md` updates — diff against the prior dated section to
|
||||
> see what's actually moved.
|
||||
|
||||
Connection: `{dsn_redacted}`
|
||||
Statement timeout: {stmt_timeout}
|
||||
"""
|
||||
|
||||
|
||||
# ─── SQL fragments ──────────────────────────────────────────────────────────
|
||||
|
||||
ROW_COUNTS_SQL = """
|
||||
SELECT 'contacts' tbl, count(*)::bigint n,
|
||||
min(qso_timestamp)::date lo, max(qso_timestamp)::date hi FROM contacts
|
||||
UNION ALL SELECT 'surface_observations', count(*),
|
||||
min(observed_at)::date, max(observed_at)::date FROM surface_observations
|
||||
UNION ALL SELECT 'soundings', count(*),
|
||||
min(observed_at)::date, max(observed_at)::date FROM soundings
|
||||
UNION ALL SELECT 'hrrr_profiles', count(*),
|
||||
min(valid_time)::date, max(valid_time)::date FROM hrrr_profiles
|
||||
UNION ALL SELECT 'hrrr_native_profiles', count(*),
|
||||
min(valid_time)::date, max(valid_time)::date FROM hrrr_native_profiles
|
||||
UNION ALL SELECT 'era5_profiles', count(*),
|
||||
min(valid_time)::date, max(valid_time)::date FROM era5_profiles
|
||||
UNION ALL SELECT 'iemre_observations', count(*), NULL::date, NULL::date
|
||||
FROM iemre_observations
|
||||
UNION ALL SELECT 'nexrad_observations', count(*),
|
||||
min(observed_at)::date, max(observed_at)::date FROM nexrad_observations
|
||||
UNION ALL SELECT 'rtma_observations', count(*),
|
||||
min(valid_time)::date, max(valid_time)::date FROM rtma_observations
|
||||
UNION ALL SELECT 'terrain_profiles', count(*), NULL::date, NULL::date
|
||||
FROM terrain_profiles
|
||||
UNION ALL SELECT 'propagation_scores', count(*),
|
||||
min(valid_time)::date, max(valid_time)::date FROM propagation_scores
|
||||
ORDER BY tbl;
|
||||
"""
|
||||
|
||||
CONTACTS_BY_BAND_SQL = """
|
||||
SELECT band::int AS band_mhz, count(*) AS contacts,
|
||||
ROUND(AVG(distance_km::numeric)) AS avg_km,
|
||||
MAX(distance_km::numeric) AS max_km
|
||||
FROM contacts
|
||||
WHERE pos1 IS NOT NULL AND distance_km IS NOT NULL
|
||||
AND distance_km < 3000 AND flagged_invalid = false AND band >= 902
|
||||
GROUP BY 1 ORDER BY 1;
|
||||
"""
|
||||
|
||||
SOUNDING_MONTHLY_SQL = """
|
||||
SELECT EXTRACT(MONTH FROM observed_at)::int AS month,
|
||||
count(*) AS soundings,
|
||||
ROUND(100.0 * count(*) FILTER (WHERE ducting_detected) / count(*), 1) AS ducting_pct,
|
||||
ROUND(AVG(min_refractivity_gradient)::numeric, 1) AS avg_min_grad,
|
||||
ROUND(AVG(precipitable_water_mm)::numeric, 1) AS avg_pwat
|
||||
FROM soundings
|
||||
WHERE min_refractivity_gradient IS NOT NULL
|
||||
GROUP BY 1 ORDER BY 1;
|
||||
"""
|
||||
|
||||
NATIVE_DUCT_SQL = """
|
||||
SELECT
|
||||
CASE WHEN best_duct_band_ghz IS NULL THEN 'none'
|
||||
WHEN best_duct_band_ghz < 5 THEN '<5 GHz'
|
||||
WHEN best_duct_band_ghz < 15 THEN '5-15 GHz'
|
||||
WHEN best_duct_band_ghz < 30 THEN '15-30 GHz'
|
||||
WHEN best_duct_band_ghz < 75 THEN '30-75 GHz'
|
||||
ELSE '75+ GHz' END AS band_bin,
|
||||
count(*) AS profiles,
|
||||
ROUND(AVG(inversion_top_m)::numeric) AS avg_inv_top_m,
|
||||
ROUND(AVG(theta_e_jump_k)::numeric, 1) AS avg_theta_e_jump,
|
||||
ROUND(AVG(bulk_richardson)::numeric, 2) AS avg_richardson
|
||||
FROM hrrr_native_profiles GROUP BY 1 ORDER BY 1;
|
||||
"""
|
||||
|
||||
# Per-band contact ↔ HRRR Pearson correlations.
|
||||
PER_BAND_JOIN_SQL = """
|
||||
WITH joined AS (
|
||||
SELECT DISTINCT ON (c.id)
|
||||
c.id, c.band::int AS band, c.distance_km::float AS dist,
|
||||
h.surface_temp_c AS tc, h.surface_dewpoint_c AS dpc,
|
||||
h.surface_pressure_mb AS pr, h.pwat_mm AS pwat,
|
||||
h.min_refractivity_gradient AS grad,
|
||||
h.surface_refractivity AS sref, h.hpbl_m AS hpbl
|
||||
FROM contacts c
|
||||
JOIN hrrr_profiles h
|
||||
ON h.lat BETWEEN (c.pos1->>'lat')::float - 0.07
|
||||
AND (c.pos1->>'lat')::float + 0.07
|
||||
AND h.lon BETWEEN (c.pos1->>'lon')::float - 0.07
|
||||
AND (c.pos1->>'lon')::float + 0.07
|
||||
AND h.valid_time BETWEEN c.qso_timestamp - INTERVAL '1 hour'
|
||||
AND c.qso_timestamp + INTERVAL '1 hour'
|
||||
WHERE c.pos1 IS NOT NULL AND c.distance_km < 3000 AND c.flagged_invalid = false
|
||||
ORDER BY c.id, ABS(EXTRACT(EPOCH FROM h.valid_time - c.qso_timestamp))
|
||||
)
|
||||
SELECT * FROM joined;
|
||||
"""
|
||||
|
||||
# Empty-table / data-quality smoke checks.
|
||||
DATA_GAP_SQL = """
|
||||
SELECT
|
||||
(SELECT count(*) FROM era5_profiles) AS era5_rows,
|
||||
(SELECT count(*) FROM hrrr_climatology) AS hrrr_climatology_rows,
|
||||
(SELECT count(*) FROM rtma_observations) AS rtma_rows,
|
||||
(SELECT count(*) FROM metar_5min_observations) AS metar5_rows,
|
||||
(SELECT count(*) FROM contacts WHERE hrrr_status = 'complete') AS hrrr_complete_contacts,
|
||||
(SELECT count(*) FROM contacts WHERE hrrr_status != 'complete') AS hrrr_pending_contacts;
|
||||
"""
|
||||
|
||||
|
||||
# ─── Reporting helpers ──────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def fetch_df(conn, sql: str) -> pd.DataFrame:
|
||||
with conn.cursor(row_factory=dict_row) as cur:
|
||||
cur.execute(sql)
|
||||
rows = cur.fetchall()
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def md_table(df: pd.DataFrame) -> str:
|
||||
if df.empty:
|
||||
return "_(no rows)_\n"
|
||||
return df.to_markdown(index=False, floatfmt=".3f") + "\n"
|
||||
|
||||
|
||||
def correlations_per_band(df: pd.DataFrame, min_samples: int = 30) -> pd.DataFrame:
|
||||
fields = ["pr", "dpc", "pwat", "sref", "grad", "tc", "hpbl"]
|
||||
rows = []
|
||||
for band, group in df.groupby("band"):
|
||||
if len(group) < min_samples:
|
||||
continue
|
||||
row = {"band_mhz": int(band), "n": len(group)}
|
||||
for f in fields:
|
||||
try:
|
||||
row[f"rho_{f}"] = round(group["dist"].corr(group[f]), 3)
|
||||
except Exception:
|
||||
row[f"rho_{f}"] = None
|
||||
rows.append(row)
|
||||
return pd.DataFrame(rows).sort_values("band_mhz")
|
||||
|
||||
|
||||
def hpbl_bins(df: pd.DataFrame, band_mhz: int = 10000) -> pd.DataFrame:
|
||||
sub = df[(df["band"] == band_mhz) & df["hpbl"].notna()].copy()
|
||||
if sub.empty:
|
||||
return pd.DataFrame()
|
||||
bins = [0, 200, 500, 1000, 1500, 2000, 1e9]
|
||||
labels = ["<200", "200-500", "500-1000", "1000-1500", "1500-2000", ">=2000"]
|
||||
sub["bin"] = pd.cut(sub["hpbl"], bins=bins, labels=labels, right=False)
|
||||
out = (
|
||||
sub.groupby("bin", observed=True)["dist"]
|
||||
.agg(["count", "mean", "median"])
|
||||
.round(1)
|
||||
.rename(columns={"count": "n", "mean": "avg_km", "median": "p50_km"})
|
||||
.reset_index()
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def pressure_bins(df: pd.DataFrame, band_mhz: int = 10000) -> pd.DataFrame:
|
||||
sub = df[(df["band"] == band_mhz) & df["pr"].notna()].copy()
|
||||
if sub.empty:
|
||||
return pd.DataFrame()
|
||||
bins = [0, 990, 1000, 1010, 1020, 9999]
|
||||
labels = ["<990", "990-1000", "1000-1010", "1010-1020", ">=1020"]
|
||||
sub["bin"] = pd.cut(sub["pr"], bins=bins, labels=labels, right=False)
|
||||
out = (
|
||||
sub.groupby("bin", observed=True)["dist"]
|
||||
.agg(["count", "mean", "median"])
|
||||
.round(1)
|
||||
.rename(columns={"count": "n", "mean": "avg_km", "median": "p50_km"})
|
||||
.reset_index()
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def redact(dsn: str) -> str:
|
||||
if "@" not in dsn:
|
||||
return dsn
|
||||
head, tail = dsn.split("@", 1)
|
||||
if "://" in head and ":" in head.split("://", 1)[1]:
|
||||
scheme, rest = head.split("://", 1)
|
||||
user = rest.split(":", 1)[0]
|
||||
return f"{scheme}://{user}:***@{tail}"
|
||||
return dsn
|
||||
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--dsn",
|
||||
default=os.environ.get("PROP_PROD_DB_URL"),
|
||||
help="Postgres connection string (defaults to $PROP_PROD_DB_URL)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--out",
|
||||
default=None,
|
||||
help="Output Markdown path (defaults to docs/algo-reports/YYYY-MM-DD-recalibration.md)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--statement-timeout",
|
||||
default="20min",
|
||||
help="Postgres statement_timeout (default: 20min)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.dsn:
|
||||
print(
|
||||
"error: PROP_PROD_DB_URL not set and --dsn not given; populate "
|
||||
"via .envrc",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
today = dt.date.today().isoformat()
|
||||
out_path = Path(
|
||||
args.out
|
||||
or Path(__file__).resolve().parent.parent
|
||||
/ "docs"
|
||||
/ "algo-reports"
|
||||
/ f"{today}-recalibration.md"
|
||||
)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print(f"connecting to {redact(args.dsn)}", file=sys.stderr)
|
||||
|
||||
sections: list[str] = []
|
||||
sections.append(
|
||||
REPORT_HEADER.format(
|
||||
today=today,
|
||||
dsn_redacted=redact(args.dsn),
|
||||
stmt_timeout=args.statement_timeout,
|
||||
)
|
||||
)
|
||||
|
||||
with psycopg.connect(args.dsn, autocommit=True) as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(f"SET statement_timeout = '{args.statement_timeout}';")
|
||||
|
||||
print("• row counts", file=sys.stderr)
|
||||
sections.append("\n## Row counts and date ranges\n")
|
||||
sections.append(md_table(fetch_df(conn, ROW_COUNTS_SQL)))
|
||||
|
||||
print("• data gaps", file=sys.stderr)
|
||||
gaps = fetch_df(conn, DATA_GAP_SQL).iloc[0].to_dict()
|
||||
sections.append("\n## Data gaps (these should NOT be zero in a healthy prod)\n")
|
||||
sections.append(
|
||||
"\n".join(f"- **{k}**: {v}" for k, v in gaps.items()) + "\n"
|
||||
)
|
||||
|
||||
print("• contacts by band", file=sys.stderr)
|
||||
sections.append("\n## Contacts by band (≥902 MHz)\n")
|
||||
sections.append(md_table(fetch_df(conn, CONTACTS_BY_BAND_SQL)))
|
||||
|
||||
print("• sounding monthly", file=sys.stderr)
|
||||
sections.append("\n## Monthly sounding ducting\n")
|
||||
sections.append(md_table(fetch_df(conn, SOUNDING_MONTHLY_SQL)))
|
||||
|
||||
print("• native HRRR ducts", file=sys.stderr)
|
||||
sections.append(
|
||||
"\n## HRRR native-profile duct distribution (best supportable band)\n"
|
||||
)
|
||||
sections.append(md_table(fetch_df(conn, NATIVE_DUCT_SQL)))
|
||||
|
||||
print("• per-band correlations (this is the slow one)", file=sys.stderr)
|
||||
joined = fetch_df(conn, PER_BAND_JOIN_SQL)
|
||||
|
||||
if joined.empty:
|
||||
sections.append("\n_No contacts ↔ HRRR matches were found._\n")
|
||||
else:
|
||||
sections.append(
|
||||
f"\n## Contact ↔ HRRR per-band Pearson correlations "
|
||||
f"(matched n={len(joined)})\n"
|
||||
)
|
||||
sections.append(md_table(correlations_per_band(joined)))
|
||||
|
||||
sections.append("\n## 10 GHz: HPBL bin distance distribution\n")
|
||||
sections.append(md_table(hpbl_bins(joined)))
|
||||
|
||||
sections.append("\n## 10 GHz: pressure bin distance distribution\n")
|
||||
sections.append(md_table(pressure_bins(joined)))
|
||||
|
||||
out_path.write_text("\n".join(sections))
|
||||
print(f"wrote {out_path}", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
|
|
@ -3,7 +3,27 @@ defmodule Microwaveprop.Propagation.BandConfigTest do
|
|||
|
||||
alias Microwaveprop.Propagation.BandConfig
|
||||
|
||||
@all_freqs [902, 1_296, 2_304, 3_456, 5_760, 10_000, 24_000, 47_000, 68_000, 75_000, 122_000, 134_000, 241_000]
|
||||
@all_freqs [
|
||||
902,
|
||||
1_296,
|
||||
2_304,
|
||||
3_456,
|
||||
5_760,
|
||||
10_000,
|
||||
24_000,
|
||||
47_000,
|
||||
68_000,
|
||||
75_000,
|
||||
122_000,
|
||||
134_000,
|
||||
142_000,
|
||||
145_000,
|
||||
241_000,
|
||||
288_000,
|
||||
322_000,
|
||||
403_000,
|
||||
411_000
|
||||
]
|
||||
|
||||
describe "get/1" do
|
||||
test "returns config for 10 GHz" do
|
||||
|
|
@ -95,12 +115,44 @@ defmodule Microwaveprop.Propagation.BandConfigTest do
|
|||
test "returns nil for unknown band" do
|
||||
assert BandConfig.get(999) == nil
|
||||
end
|
||||
|
||||
test "returns config for 142 GHz (between H2O 118 and 183 lines)" do
|
||||
config = BandConfig.get(142_000)
|
||||
assert config.freq_mhz == 142_000
|
||||
assert config.humidity_effect == :harmful
|
||||
assert config.rain_alpha < 1.0
|
||||
end
|
||||
|
||||
test "returns config for 145 GHz" do
|
||||
config = BandConfig.get(145_000)
|
||||
assert config.freq_mhz == 145_000
|
||||
assert config.humidity_effect == :harmful
|
||||
end
|
||||
|
||||
test "returns config for 288 GHz (sub-mm window)" do
|
||||
config = BandConfig.get(288_000)
|
||||
assert config.freq_mhz == 288_000
|
||||
assert config.humidity_effect == :harmful
|
||||
assert config.typical_range_km <= 10
|
||||
end
|
||||
|
||||
test "returns config for 322 GHz" do
|
||||
assert %{freq_mhz: 322_000, humidity_effect: :harmful} = BandConfig.get(322_000)
|
||||
end
|
||||
|
||||
test "returns config for 403 GHz" do
|
||||
assert %{freq_mhz: 403_000, humidity_effect: :harmful} = BandConfig.get(403_000)
|
||||
end
|
||||
|
||||
test "returns config for 411 GHz" do
|
||||
assert %{freq_mhz: 411_000, humidity_effect: :harmful} = BandConfig.get(411_000)
|
||||
end
|
||||
end
|
||||
|
||||
describe "all_bands/0" do
|
||||
test "returns 13 bands" do
|
||||
test "returns 19 bands (13 original + 142, 145, 288, 322, 403, 411 GHz)" do
|
||||
bands = BandConfig.all_bands()
|
||||
assert length(bands) == 13
|
||||
assert length(bands) == 19
|
||||
end
|
||||
|
||||
test "bands are sorted by frequency" do
|
||||
|
|
|
|||
|
|
@ -232,29 +232,56 @@ defmodule Microwaveprop.Propagation.ScorerTest do
|
|||
# ── score_refractivity/3 ─────────────────────────────────────────
|
||||
|
||||
describe "score_refractivity/3" do
|
||||
# 750 m sits in the [500, 1500) baseline-multiplier band so existing
|
||||
# threshold assertions don't pick up the HPBL adjustment.
|
||||
@baseline_bl 750
|
||||
|
||||
test "strong ducting gradient returns high score for beneficial" do
|
||||
# gradient < -500 -> 98 for beneficial
|
||||
assert Scorer.score_refractivity(-600, 500, @band_10g) == 98
|
||||
assert Scorer.score_refractivity(-600, @baseline_bl, @band_10g) == 98
|
||||
end
|
||||
|
||||
test "strong gradient returns 85 for harmful" do
|
||||
# gradient < -200 -> 85 for harmful (HRRR-calibrated)
|
||||
assert Scorer.score_refractivity(-250, 500, @band_24g) == 85
|
||||
assert Scorer.score_refractivity(-250, @baseline_bl, @band_24g) == 85
|
||||
end
|
||||
|
||||
test "moderate gradient returns appropriate score" do
|
||||
# gradient < -150 but >= -200 -> 92 for beneficial (HRRR-calibrated)
|
||||
assert Scorer.score_refractivity(-160, 500, @band_10g) == 92
|
||||
assert Scorer.score_refractivity(-160, @baseline_bl, @band_10g) == 92
|
||||
end
|
||||
|
||||
test "nil gradient returns 50" do
|
||||
assert Scorer.score_refractivity(nil, 500, @band_10g) == 50
|
||||
assert Scorer.score_refractivity(nil, @baseline_bl, @band_10g) == 50
|
||||
end
|
||||
|
||||
test "weak gradient returns default regardless of BL depth" do
|
||||
# gradient > -40 (no threshold match) -> default 42
|
||||
assert Scorer.score_refractivity(-30, 200, @band_10g) == 42
|
||||
assert Scorer.score_refractivity(-30, 500, @band_10g) == 42
|
||||
test "nil bl_depth_m treated as baseline (no HPBL multiplier)" do
|
||||
# A nil HPBL must not crash and must produce the unmultiplied score.
|
||||
assert Scorer.score_refractivity(-160, nil, @band_10g) == 92
|
||||
end
|
||||
|
||||
test "shallow boundary layer (<200 m) boosts the gradient score" do
|
||||
# Same -100 gradient: shallow BL should clearly out-score a baseline BL.
|
||||
shallow = Scorer.score_refractivity(-100, 150, @band_10g)
|
||||
baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g)
|
||||
assert shallow > baseline
|
||||
assert shallow <= 100
|
||||
end
|
||||
|
||||
test "deep boundary layer (>=2000 m) penalises the gradient score" do
|
||||
# Same -100 gradient: deep BL drags the score down.
|
||||
deep = Scorer.score_refractivity(-100, 2200, @band_10g)
|
||||
baseline = Scorer.score_refractivity(-100, @baseline_bl, @band_10g)
|
||||
assert deep < baseline
|
||||
assert deep >= 0
|
||||
end
|
||||
|
||||
test "shallow BL also lifts the weak-gradient default" do
|
||||
# The HPBL multiplier applies to the default fallback too — the binned
|
||||
# data shows shallow-BL paths run ~230 km even when HRRR gradient is weak.
|
||||
shallow_default = Scorer.score_refractivity(-30, 150, @band_10g)
|
||||
baseline_default = Scorer.score_refractivity(-30, @baseline_bl, @band_10g)
|
||||
assert shallow_default > baseline_default
|
||||
end
|
||||
end
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ defmodule Microwaveprop.PropagationTest do
|
|||
valid_time = ~U[2026-07-15 13:00:00Z]
|
||||
results = Propagation.score_grid_point(hrrr_profile, valid_time, 33.0, -97.0)
|
||||
|
||||
assert length(results) == 13
|
||||
assert length(results) == 19
|
||||
|
||||
Enum.each(results, fn result ->
|
||||
assert result.score >= 0 and result.score <= 100
|
||||
|
|
|
|||
|
|
@ -3,6 +3,12 @@ defmodule Microwaveprop.Workers.Era5MonthBatchWorkerTest do
|
|||
|
||||
alias Microwaveprop.Workers.Era5MonthBatchWorker
|
||||
|
||||
describe "worker config" do
|
||||
test "max_attempts bumped to 5 so CDS slowness doesn't discard tiles after 3 tries" do
|
||||
assert Era5MonthBatchWorker.__opts__()[:max_attempts] == 5
|
||||
end
|
||||
end
|
||||
|
||||
describe "unique constraint" do
|
||||
test "collapses identical (year, month, tile_lat, tile_lon) jobs" do
|
||||
args = %{year: 2014, month: 3, tile_lat: 32, tile_lon: -98}
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue