diff --git a/docs/algo-reports/2026-05-04-recalibration.md b/docs/algo-reports/2026-05-04-recalibration.md new file mode 100644 index 00000000..95623ff3 --- /dev/null +++ b/docs/algo-reports/2026-05-04-recalibration.md @@ -0,0 +1,552 @@ +# Recalibration Analysis Report — 2026-05-04 + +> Auto-generated by `scripts/recalibrate_algo.py`. This report is the +> input for `algo.md` updates — diff against the prior dated section to +> see what's actually moved. + +Connection: `postgres://prop:***@10.0.15.30:5432/prop` +Statement timeout: 20min + + +## Row counts and date ranges + +| tbl | n | lo | hi | +|:---------------------|-------------:|:-----------|:-----------| +| contacts | 82084.000 | 1991-05-04 | 2026-05-02 | +| hrrr_native_profiles | 11472.000 | 2019-03-09 | 2025-09-21 | +| hrrr_profiles | 81302322.000 | 2016-09-17 | 2026-05-02 | +| iemre_observations | 28936.000 | | | +| narr_profiles | 395.000 | 1991-05-04 | 2014-09-19 | +| nexrad_observations | 7286.000 | 2019-08-17 | 2024-09-22 | +| propagation_scores | nan | | | +| soundings | 27745.000 | 1990-06-07 | 2026-05-04 | +| surface_observations | 648809.000 | 1991-05-03 | 2026-05-04 | +| terrain_profiles | 82084.000 | | | + + +## Data gaps and ingestion health + +Severity legend: **BROKEN** = table is empty and feeds an active scoring path; **WARN** = populated but coverage is low enough to skew per-band statistics; **OK** = within spec or informational. Each non-OK row carries a one-line remediation hint. + +| metric | value | severity | remediation | +|:-----------------------|--------:|:-----------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| narr_rows | 395 | OK | | +| narr_pre2014_rows | 395 | OK | | +| hrrr_climatology_rows | 0 | BROKEN | Should self-heal within 24h via the daily 02:30Z cron that runs AdminTaskWorker(task=climatology). If still 0 after a day, check Oban dashboard for failed admin jobs. | +| hrrr_complete_contacts | 81278 | OK | | +| hrrr_pending_contacts | 806 | OK | | +| pre2014_contacts | 207 | OK | | + + +HRRR enrichment completeness: **99.0%** (81,278 complete / 806 pending). + + +### NARR coverage by year + +Year-bucketed counts. NARR is supposed to span 1979 → 2014-10 at 3-hourly cadence (~2,920 timestamps per year per grid cell). Years far below the others are partial backfills the worker hasn't finished. + +| year | profiles | +|---------:|-----------:| +| 1991.000 | 31.000 | +| 1992.000 | 4.000 | +| 1993.000 | 18.000 | +| 1994.000 | 19.000 | +| 1996.000 | 3.000 | +| 1997.000 | 8.000 | +| 1998.000 | 2.000 | +| 1999.000 | 11.000 | +| 2000.000 | 10.000 | +| 2001.000 | 14.000 | +| 2002.000 | 12.000 | +| 2003.000 | 36.000 | +| 2004.000 | 23.000 | +| 2005.000 | 49.000 | +| 2006.000 | 20.000 | +| 2007.000 | 44.000 | +| 2008.000 | 15.000 | +| 2009.000 | 17.000 | +| 2010.000 | 31.000 | +| 2011.000 | 4.000 | +| 2012.000 | 15.000 | +| 2013.000 | 3.000 | +| 2014.000 | 6.000 | + + +### HRRR enrichment status by year + +Locate the pending backlog: a year with high `queued`/`failed`/`unset` counts is where HrrrFetchWorker needs to be rerun. Recent years dominating `queued` usually means the live Oban queue is stuck rather than a historical backfill gap. + +| year | complete | queued | failed | unset | +|---------:|-----------:|---------:|---------:|--------:| +| 2014.000 | 0.000 | 0.000 | 0.000 | 0.000 | +| 2015.000 | 0.000 | 0.000 | 0.000 | 0.000 | +| 2016.000 | 2.000 | 0.000 | 0.000 | 0.000 | +| 2017.000 | 26.000 | 0.000 | 0.000 | 0.000 | +| 2018.000 | 2658.000 | 0.000 | 0.000 | 0.000 | +| 2019.000 | 13036.000 | 0.000 | 0.000 | 0.000 | +| 2020.000 | 10300.000 | 0.000 | 0.000 | 0.000 | +| 2021.000 | 11266.000 | 0.000 | 0.000 | 0.000 | +| 2022.000 | 14487.000 | 2.000 | 0.000 | 0.000 | +| 2023.000 | 13461.000 | 5.000 | 0.000 | 0.000 | +| 2024.000 | 13378.000 | 0.000 | 0.000 | 0.000 | +| 2025.000 | 2640.000 | 0.000 | 0.000 | 0.000 | +| 2026.000 | 24.000 | 0.000 | 0.000 | 0.000 | + + +## Contacts by band (≥50 MHz) + +| band_mhz | contacts | avg_km | max_km | +|-----------:|-----------:|---------:|---------:| +| 222 | 7595 | 229.000 | 2306.000 | +| 432 | 9177 | 200.000 | 1752.000 | +| 902 | 1794 | 171.000 | 915.000 | +| 1296 | 2996 | 160.000 | 1752.000 | +| 2304 | 749 | 153.000 | 655.000 | +| 3400 | 351 | 151.000 | 643.000 | +| 5760 | 336 | 126.000 | 505.000 | +| 10000 | 54280 | 211.000 | 2393.000 | +| 24000 | 3815 | 96.000 | 710.000 | +| 47000 | 774 | 65.000 | 343.000 | +| 75000 | 117 | 57.000 | 289.000 | +| 122000 | 70 | 18.000 | 139.000 | +| 134000 | 5 | 65.000 | 157.000 | +| 142000 | 4 | 47.000 | 79.700 | +| 145000 | 2 | 40.000 | 79.600 | +| 241000 | 14 | 31.000 | 114.400 | +| 288000 | 1 | 1.000 | 1.246 | +| 322000 | 1 | 1.000 | 1.400 | +| 403000 | 1 | 1.000 | 1.400 | +| 411000 | 1 | 0.000 | 0.050 | + + +## Monthly sounding ducting + +| month | soundings | ducting_pct | avg_min_grad | avg_pwat | +|--------:|------------:|--------------:|---------------:|-----------:| +| 1 | 103 | 27.200 | -179.200 | 10.400 | +| 2 | 131 | 29.000 | -163.100 | 7.900 | +| 3 | 93 | 17.200 | -157.200 | 8.900 | +| 4 | 330 | 24.200 | -155.500 | 11.300 | +| 5 | 360 | 48.100 | -269.200 | 23.900 | +| 6 | 396 | 68.900 | -352.900 | 31.600 | +| 7 | 304 | 67.100 | -292.500 | 29.600 | +| 8 | 5045 | 55.200 | -255.800 | 35.100 | +| 9 | 2345 | 56.300 | -280.000 | 27.300 | +| 10 | 179 | 55.300 | -304.800 | 18.300 | +| 11 | 194 | 45.400 | -237.300 | 11.600 | +| 12 | 158 | 25.900 | -182.400 | 10.800 | + + +## HRRR native-profile duct distribution (best supportable band) + +| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson | +|:-----------|-----------:|----------------:|-------------------:|-----------------:| +| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 | +| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 | +| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 | +| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 | +| none | 9928 | 11497.000 | 80.500 | 38.270 | + + +## Contact ↔ HRRR per-band Pearson correlations (matched n=81547) + +| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl | +|-----------:|----------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:| +| 222.000 | 7518.000 | -0.014 | 0.077 | 0.068 | 0.087 | 0.040 | -0.081 | -0.039 | +| 432.000 | 9073.000 | 0.074 | 0.134 | 0.111 | 0.132 | 0.008 | -0.088 | -0.063 | +| 902.000 | 1754.000 | 0.080 | 0.097 | 0.050 | 0.090 | -0.039 | -0.070 | -0.029 | +| 1296.000 | 2951.000 | 0.025 | 0.103 | 0.053 | 0.110 | -0.077 | -0.041 | -0.051 | +| 2304.000 | 719.000 | 0.079 | 0.165 | 0.130 | 0.192 | -0.126 | -0.059 | -0.086 | +| 3400.000 | 335.000 | -0.026 | 0.154 | 0.150 | 0.221 | 0.061 | -0.061 | -0.024 | +| 5760.000 | 320.000 | 0.156 | 0.076 | 0.099 | 0.091 | 0.003 | -0.068 | -0.057 | +| 10000.000 | 54150.000 | -0.086 | -0.022 | -0.029 | 0.000 | 0.030 | -0.004 | 0.007 | +| 24000.000 | 3794.000 | -0.221 | -0.338 | -0.362 | -0.241 | 0.023 | -0.211 | -0.009 | +| 47000.000 | 757.000 | -0.256 | -0.215 | -0.233 | -0.054 | -0.021 | -0.113 | -0.079 | +| 75000.000 | 109.000 | -0.768 | -0.699 | -0.620 | -0.582 | 0.434 | -0.680 | 0.365 | +| 122000.000 | 63.000 | -0.041 | -0.129 | -0.127 | -0.248 | 0.355 | 0.039 | 0.400 | + + +## Per-band HPBL bin distance distribution + +One table per band with ≥50 matched contacts. A band-specific effect here means HPBL belongs in the scorer for that band. + + +**222 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 1224 | 266.100 | 205.500 | +| 200-500 | 1556 | 225.600 | 167.000 | +| 500-1000 | 1754 | 213.300 | 167.500 | +| 1000-1500 | 1426 | 223.700 | 170.000 | +| 1500-2000 | 1239 | 224.700 | 178.000 | +| >=2000 | 319 | 233.800 | 158.000 | + + +**432 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 1579 | 224.500 | 178.000 | +| 200-500 | 1867 | 205.800 | 153.000 | +| 500-1000 | 2114 | 189.900 | 139.800 | +| 1000-1500 | 1683 | 196.800 | 148.000 | +| 1500-2000 | 1475 | 192.600 | 145.000 | +| >=2000 | 355 | 171.700 | 124.000 | + + +**902 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 271 | 198.300 | 162.000 | +| 200-500 | 366 | 164.900 | 138.000 | +| 500-1000 | 433 | 166.500 | 131.600 | +| 1000-1500 | 356 | 166.800 | 136.500 | +| 1500-2000 | 282 | 174.900 | 152.000 | +| >=2000 | 46 | 153.500 | 126.500 | + + +**1.296 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 443 | 174.500 | 145.000 | +| 200-500 | 612 | 173.100 | 133.500 | +| 500-1000 | 735 | 152.500 | 116.000 | +| 1000-1500 | 585 | 158.200 | 131.000 | +| 1500-2000 | 479 | 152.600 | 122.000 | +| >=2000 | 97 | 143.800 | 108.000 | + + +**2.304 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 101 | 167.000 | 151.000 | +| 200-500 | 131 | 182.200 | 157.000 | +| 500-1000 | 188 | 140.800 | 122.000 | +| 1000-1500 | 159 | 146.700 | 135.000 | +| 1500-2000 | 123 | 148.300 | 135.000 | +| >=2000 | 17 | 137.400 | 140.000 | + + +**3.4 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 54 | 143.100 | 122.000 | +| 200-500 | 60 | 178.200 | 141.000 | +| 500-1000 | 74 | 145.600 | 108.000 | +| 1000-1500 | 84 | 149.400 | 133.000 | +| 1500-2000 | 48 | 165.100 | 138.700 | +| >=2000 | 15 | 121.800 | 128.000 | + + +**5.76 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 37 | 148.800 | 108.000 | +| 200-500 | 54 | 138.400 | 135.000 | +| 500-1000 | 85 | 113.300 | 108.000 | +| 1000-1500 | 74 | 130.900 | 127.500 | +| 1500-2000 | 54 | 121.200 | 122.000 | +| >=2000 | 16 | 124.200 | 125.000 | + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|------:|---------:|---------:| +| <200 | 10466 | 212.500 | 182.600 | +| 200-500 | 14330 | 205.900 | 179.500 | +| 500-1000 | 16463 | 208.700 | 190.300 | +| 1000-1500 | 8712 | 212.400 | 198.500 | +| 1500-2000 | 2985 | 201.200 | 179.000 | +| >=2000 | 1191 | 228.700 | 193.500 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <200 | 731 | 100.200 | 92.200 | +| 200-500 | 1079 | 91.500 | 87.700 | +| 500-1000 | 1179 | 94.800 | 87.700 | +| 1000-1500 | 534 | 98.500 | 95.700 | +| 1500-2000 | 196 | 99.900 | 99.400 | +| >=2000 | 75 | 74.300 | 31.800 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 172 | 67.100 | 63.600 | +| 200-500 | 237 | 62.600 | 61.300 | +| 500-1000 | 253 | 61.900 | 65.400 | +| 1000-1500 | 58 | 71.000 | 70.900 | +| 1500-2000 | 29 | 33.500 | 23.700 | +| >=2000 | 8 | 33.900 | 32.300 | + + +**75 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 19 | 18.600 | 19.600 | +| 200-500 | 31 | 44.300 | 36.700 | +| 500-1000 | 53 | 58.800 | 36.700 | +| 1000-1500 | 6 | 56.100 | 29.800 | + + +**122 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <200 | 7 | 4.400 | 5.000 | +| 200-500 | 27 | 8.800 | 8.000 | +| 500-1000 | 18 | 13.600 | 7.000 | +| 1000-1500 | 6 | 78.500 | 85.500 | +| 1500-2000 | 3 | 10.000 | 12.000 | +| >=2000 | 2 | 5.000 | 5.000 | + + +## Per-band pressure bin distance distribution + +Surface pressure tends to proxy synoptic-scale stagnation, so we want to see longer distances under the 1015-1025 mb ridge at every band that ducts. + + +**222 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 2720 | 227.200 | 178.000 | +| 990-1000 | 1926 | 255.100 | 201.000 | +| 1000-1010 | 1799 | 233.800 | 186.000 | +| 1010-1020 | 953 | 181.400 | 131.000 | +| >=1020 | 120 | 166.100 | 132.000 | + + +**432 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 3536 | 188.800 | 141.000 | +| 990-1000 | 2278 | 226.700 | 178.000 | +| 1000-1010 | 1832 | 210.900 | 156.500 | +| 1010-1020 | 1264 | 174.100 | 128.000 | +| >=1020 | 163 | 159.000 | 121.000 | + + +**902 MHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 590 | 154.100 | 127.000 | +| 990-1000 | 510 | 203.200 | 157.500 | +| 1000-1010 | 369 | 189.200 | 148.000 | +| 1010-1020 | 249 | 139.200 | 143.000 | +| >=1020 | 36 | 79.700 | 51.500 | + + +**1.296 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 1038 | 155.400 | 125.000 | +| 990-1000 | 800 | 183.100 | 146.000 | +| 1000-1010 | 620 | 165.400 | 117.000 | +| 1010-1020 | 444 | 132.400 | 114.000 | +| >=1020 | 49 | 119.800 | 101.000 | + + +**2.304 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 190 | 146.500 | 134.000 | +| 990-1000 | 250 | 155.000 | 122.000 | +| 1000-1010 | 144 | 172.600 | 134.000 | +| 1010-1020 | 121 | 147.600 | 152.000 | +| >=1020 | 14 | 127.600 | 137.000 | + + +**3.4 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 95 | 154.400 | 136.000 | +| 990-1000 | 159 | 159.200 | 134.000 | +| 1000-1010 | 45 | 166.000 | 140.000 | +| 1010-1020 | 27 | 111.700 | 98.000 | +| >=1020 | 9 | 114.600 | 131.000 | + + +**5.76 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 86 | 106.000 | 105.500 | +| 990-1000 | 139 | 133.200 | 111.000 | +| 1000-1010 | 51 | 136.100 | 104.000 | +| 1010-1020 | 34 | 143.200 | 152.000 | +| >=1020 | 10 | 140.600 | 139.000 | + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|------:|---------:|---------:| +| <990 | 27142 | 221.800 | 213.500 | +| 990-1000 | 13469 | 181.200 | 158.600 | +| 1000-1010 | 7236 | 204.000 | 175.200 | +| 1010-1020 | 4658 | 218.700 | 175.300 | +| >=1020 | 1645 | 229.300 | 207.600 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|-----:|---------:|---------:| +| <990 | 1485 | 100.900 | 96.500 | +| 990-1000 | 1462 | 87.200 | 81.000 | +| 1000-1010 | 463 | 114.400 | 113.400 | +| 1010-1020 | 258 | 70.900 | 72.100 | +| >=1020 | 126 | 102.700 | 89.600 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 279 | 68.500 | 70.900 | +| 990-1000 | 297 | 58.000 | 57.300 | +| 1000-1010 | 63 | 69.600 | 68.000 | +| 1010-1020 | 89 | 60.600 | 65.400 | +| >=1020 | 29 | 43.400 | 42.000 | + + +**75 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 55 | 65.300 | 57.300 | +| 990-1000 | 32 | 17.500 | 10.400 | +| 1000-1010 | 12 | 71.800 | 89.800 | +| 1010-1020 | 8 | 18.300 | 19.600 | +| >=1020 | 2 | 11.700 | 11.700 | + + +**122 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|----:|---------:|---------:| +| <990 | 22 | 12.000 | 7.000 | +| 990-1000 | 19 | 27.300 | 7.000 | +| 1000-1010 | 10 | 8.400 | 6.000 | +| 1010-1020 | 10 | 13.400 | 8.000 | +| >=1020 | 2 | 12.000 | 12.000 | + + +## Contact ↔ NARR per-band Pearson correlations (pre-2014 only, matched n=201) + +Historical sanity check: if the HRRR-era signs and magnitudes survive on 30+ years of NARR reanalysis, the scoring factors are physical rather than HRRR-artefact. Diverging signs = flag it. + +| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl | +|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:| +| 10000.000 | 103.000 | -0.402 | -0.402 | -0.402 | -0.234 | -0.222 | -0.402 | -0.402 | + + +## NEXRAD composite reflectivity vs distance (matched n=20525) + +Rain-attenuation sanity check — one table per band with ≥50 matched contacts. At rain-sensitive bands (24+ GHz) higher `max_dbz` should map to shorter contacts; at 10 GHz and below the effect should be barely detectable. + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|------:|---------:|---------:| +| none (<5) | 13803 | 207.900 | 193.500 | +| drizzle (5-20) | 1270 | 201.800 | 200.400 | +| light (20-30) | 1696 | 209.600 | 187.900 | +| moderate (30-45) | 1594 | 229.500 | 213.900 | +| heavy (>=45) | 675 | 191.600 | 146.300 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|----:|---------:|---------:| +| none (<5) | 868 | 91.500 | 83.700 | +| drizzle (5-20) | 74 | 108.300 | 123.200 | +| light (20-30) | 151 | 88.200 | 70.900 | +| moderate (30-45) | 76 | 109.300 | 122.400 | +| heavy (>=45) | 49 | 86.600 | 102.300 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:-----------------|----:|---------:|---------:| +| none (<5) | 186 | 52.400 | 56.000 | +| light (20-30) | 30 | 59.300 | 66.300 | +| moderate (30-45) | 9 | 75.100 | 81.600 | +| heavy (>=45) | 15 | 71.400 | 82.800 | + + +## hrrr_native_profiles.best_duct_band_ghz vs distance (matched n=56845) + +Validates the 1.15× boost in `Scorer.score_refractivity/4`: contacts where the native duct supports the target band should run longer than those where it does not. One table per band with ≥50 matched contacts. + + +**10 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|------:|---------:|---------:| +| none | 44655 | 211.200 | 192.400 | +| <5 GHz | 7073 | 200.000 | 169.300 | +| 5-10 GHz | 439 | 199.000 | 174.200 | +| 10-24 GHz | 146 | 204.700 | 242.700 | +| 24-47 GHz | 5 | 176.600 | 133.300 | +| >=47 GHz | 23 | 147.800 | 131.500 | + + +**24 GHz:** + +| bin | n | avg_km | p50_km | +|:----------|---------:|---------:|---------:| +| none | 3183.000 | 94.800 | 92.200 | +| <5 GHz | 473.000 | 98.600 | 84.900 | +| 5-10 GHz | 36.000 | 97.500 | 113.700 | +| 10-24 GHz | 2.000 | 44.700 | 44.700 | +| >=47 GHz | 6.000 | 131.500 | 131.500 | + + +**47 GHz:** + +| bin | n | avg_km | p50_km | +|:---------|--------:|---------:|---------:| +| none | 552.000 | 64.300 | 65.400 | +| <5 GHz | 131.000 | 56.800 | 61.800 | +| 5-10 GHz | 6.000 | 98.100 | 113.700 | + + +**75 GHz:** + +| bin | n | avg_km | p50_km | +|:-------|-------:|---------:|---------:| +| none | 77.000 | 54.200 | 36.700 | +| <5 GHz | 15.000 | 13.900 | 14.600 | + + +## Commercial-link rx_power degradation vs contemporaneous DFW contacts (matched n=12) + +Pearson(degradation_db, distance_km) = **0.318** + + +| bin | n | avg_km | p50_km | +|:---------------------|----:|---------:|---------:| +| negative/0 (no fade) | 10 | 134.500 | 122.500 | +| 3 dB noise floor | 2 | 119.500 | 119.500 | diff --git a/lib/microwaveprop/commercial/snmp_client.ex b/lib/microwaveprop/commercial/snmp_client.ex index be4c5548..6601accc 100644 --- a/lib/microwaveprop/commercial/snmp_client.ex +++ b/lib/microwaveprop/commercial/snmp_client.ex @@ -132,6 +132,11 @@ defmodule Microwaveprop.Commercial.SnmpClient do end defp run_cmd(cmd, args) do + runner = Application.get_env(:microwaveprop, :snmp_runner, &default_run_cmd/2) + runner.(cmd, args) + end + + defp default_run_cmd(cmd, args) do System.cmd(cmd, args, stderr_to_stdout: true) rescue ErlangError -> {"command not found: #{cmd}", 127} diff --git a/mix.exs b/mix.exs index 79fd9ba6..387b4b42 100644 --- a/mix.exs +++ b/mix.exs @@ -120,6 +120,7 @@ defmodule Microwaveprop.MixProject do # of either emqtt (transitive quicer + cmake) or tortoise311 # (gen_state_machine + retry/inflight machinery we don't use). {:ecto_psql_extras, "~> 0.8"}, + {:mox, "~> 1.0", only: :test}, {:dialyxir, "~> 1.4", only: [:dev, :test], runtime: false}, {:live_table, "~> 0.4.1"}, # Transitive dep of live_table (via LiveTable.TableComponent's diff --git a/mix.lock b/mix.lock index 3dd0c4a1..3ce09ab1 100644 --- a/mix.lock +++ b/mix.lock @@ -36,9 +36,11 @@ "logger_json": {:hex, :logger_json, "7.0.4", "e315f2b9a755504658a745f3eab90d88d2cd7ac2ecfd08c8da94d8893965ab5c", [:mix], [{:decimal, ">= 0.0.0", [hex: :decimal, repo: "hexpm", optional: true]}, {:ecto, "~> 3.11", [hex: :ecto, repo: "hexpm", optional: true]}, {:jason, "~> 1.4", [hex: :jason, repo: "hexpm", optional: true]}, {:plug, "~> 1.15", [hex: :plug, repo: "hexpm", optional: true]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: true]}], "hexpm", "d1369f8094e372db45d50672c3b91e8888bcd695fdc444a37a0734e96717c45c"}, "mime": {:hex, :mime, "2.0.7", "b8d739037be7cd402aee1ba0306edfdef982687ee7e9859bee6198c1e7e2f128", [:mix], [], "hexpm", "6171188e399ee16023ffc5b76ce445eb6d9672e2e241d2df6050f3c771e80ccd"}, "mint": {:hex, :mint, "1.7.1", "113fdb2b2f3b59e47c7955971854641c61f378549d73e829e1768de90fc1abf1", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "fceba0a4d0f24301ddee3024ae116df1c3f4bb7a563a731f45fdfeb9d39a231b"}, + "mox": {:hex, :mox, "1.2.0", "a2cd96b4b80a3883e3100a221e8adc1b98e4c3a332a8fc434c39526babafd5b3", [:mix], [{:nimble_ownership, "~> 1.0", [hex: :nimble_ownership, repo: "hexpm", optional: false]}], "hexpm", "c7b92b3cc69ee24a7eeeaf944cd7be22013c52fcb580c1f33f50845ec821089a"}, "msgpax": {:hex, :msgpax, "2.4.0", "4647575c87cb0c43b93266438242c21f71f196cafa268f45f91498541148c15d", [:mix], [{:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "ca933891b0e7075701a17507c61642bf6e0407bb244040d5d0a58597a06369d2"}, "nimble_csv": {:hex, :nimble_csv, "1.3.0", "b7f998dc62b222bce9596e46f028c7a5af04cb5dde6df2ea197c583227c54971", [:mix], [], "hexpm", "41ccdc18f7c8f8bb06e84164fc51635321e80d5a3b450761c4997d620925d619"}, "nimble_options": {:hex, :nimble_options, "1.1.1", "e3a492d54d85fc3fd7c5baf411d9d2852922f66e69476317787a7b2bb000a61b", [:mix], [], "hexpm", "821b2470ca9442c4b6984882fe9bb0389371b8ddec4d45a9504f00a66f650b44"}, + "nimble_ownership": {:hex, :nimble_ownership, "1.0.2", "fa8a6f2d8c592ad4d79b2ca617473c6aefd5869abfa02563a77682038bf916cf", [:mix], [], "hexpm", "098af64e1f6f8609c6672127cfe9e9590a5d3fcdd82bc17a377b8692fd81a879"}, "nimble_pool": {:hex, :nimble_pool, "1.1.0", "bf9c29fbdcba3564a8b800d1eeb5a3c58f36e1e11d7b7fb2e084a643f645f06b", [:mix], [], "hexpm", "af2e4e6b34197db81f7aad230c1118eac993acc0dae6bc83bac0126d4ae0813a"}, "nx": {:hex, :nx, "0.11.0", "d37723dbd6cfa274a5def6d6664f5680c32e2eb8a1ce25ec6d91751967fa0abf", [:mix], [{:complex, "~> 0.6", [hex: :complex, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4.0 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "36157b21239aeb251d6cbac23eb0eb3495a5e1e0cbc2e6df16afd2ede1575205"}, "oban": {:hex, :oban, "2.22.1", "9d2a38cec95070b31c1e274fae55f3925089f62159d8f3facabee0454d55b257", [:mix], [{:ecto_sql, "~> 3.10", [hex: :ecto_sql, repo: "hexpm", optional: false]}, {:ecto_sqlite3, "~> 0.9", [hex: :ecto_sqlite3, repo: "hexpm", optional: true]}, {:igniter, "~> 0.5", [hex: :igniter, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:myxql, "~> 0.7", [hex: :myxql, repo: "hexpm", optional: true]}, {:postgrex, "~> 0.20", [hex: :postgrex, repo: "hexpm", optional: true]}, {:telemetry, "~> 1.3", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "af2508c156c5b0ec30b21b0883babf7e2716af35ed5d264095896103fe3cea37"}, diff --git a/test/microwaveprop/buildings/index_test.exs b/test/microwaveprop/buildings/index_test.exs index 19767448..4ff5a52b 100644 --- a/test/microwaveprop/buildings/index_test.exs +++ b/test/microwaveprop/buildings/index_test.exs @@ -30,4 +30,35 @@ defmodule Microwaveprop.Buildings.IndexTest do assert Index.max_height_near(32.8000, -97.0000, 100) == 0.0 end + + test "records_near/3 returns matching records" do + Index.insert_records([ + %{centroid_lat: 32.8000, centroid_lon: -97.0000, max_radius_m: 10.0, height_m: 8.0}, + %{centroid_lat: 32.8001, centroid_lon: -97.0001, max_radius_m: 10.0, height_m: 25.0} + ]) + + records = Index.records_near(32.8000, -97.0000, 50) + assert length(records) == 2 + heights = Enum.map(records, & &1.height_m) |> Enum.sort() + assert heights == [8.0, 25.0] + end + + test "records_near/3 returns empty list outside radius" do + Index.insert_records([ + %{centroid_lat: 33.0, centroid_lon: -97.5, max_radius_m: 10.0, height_m: 50.0} + ]) + + assert Index.records_near(32.8, -97.0, 100) == [] + end + + test "mark_loaded/1 and loaded?/1 track loaded quadkeys" do + refute Index.loaded?("023112330") + Index.mark_loaded("023112330") + assert Index.loaded?("023112330") + end + + test "insert_records/1 handles empty list" do + Index.insert_records([]) + assert Index.max_height_near(32.8, -97.0, 100) == 0.0 + end end diff --git a/test/microwaveprop/buildings/loader_test.exs b/test/microwaveprop/buildings/loader_test.exs new file mode 100644 index 00000000..0def1d4c --- /dev/null +++ b/test/microwaveprop/buildings/loader_test.exs @@ -0,0 +1,61 @@ +defmodule Microwaveprop.Buildings.LoaderTest do + use ExUnit.Case, async: false + + alias Microwaveprop.Buildings.Index + alias Microwaveprop.Buildings.Loader + + setup do + Index.clear() + + tmp = Path.join(System.tmp_dir!(), "buildings_cache_#{System.unique_integer([:positive])}") + prev = Application.get_env(:microwaveprop, :buildings_cache_dir) + Application.put_env(:microwaveprop, :buildings_cache_dir, tmp) + + on_exit(fn -> + File.rm_rf(tmp) + if prev, do: Application.put_env(:microwaveprop, :buildings_cache_dir, prev), + else: Application.delete_env(:microwaveprop, :buildings_cache_dir) + end) + + :ok + end + + describe "ensure_loaded/1" do + test "skips loading when quadkey is already loaded" do + Index.mark_loaded("023112330") + assert Loader.ensure_loaded("023112330") == :ok + end + + test "marks quadkey as loaded when tile file doesn't exist" do + assert Loader.ensure_loaded("nonexistent_quadkey") == :ok + assert Index.loaded?("nonexistent_quadkey") + end + + test "loads records from a real tile file" do + quadkey = "023112330" + cache_dir = Application.get_env(:microwaveprop, :buildings_cache_dir) + File.mkdir_p!(cache_dir) + + geojsonl = + ~s|{"type": "Feature", "properties": {"height": 5.5}, "geometry": {"type": "Polygon", "coordinates": [[[-97.736, 32.221], [-97.736, 32.222], [-97.735, 32.222], [-97.735, 32.221], [-97.736, 32.221]]]}}| + |> then(&:zlib.gzip/1) + + File.write!(Path.join(cache_dir, "#{quadkey}.csv.gz"), geojsonl) + + assert Loader.ensure_loaded(quadkey) == :ok + assert Index.loaded?(quadkey) + assert Index.max_height_near(32.2215, -97.7355, 100) == 5.5 + end + end + + describe "ensure_loaded_for_bbox/1" do + test "returns list of quadkeys for the bbox" do + bbox = %{"south" => 32.5, "north" => 33.0, "west" => -97.5, "east" => -96.5} + keys = Loader.ensure_loaded_for_bbox(bbox) + + assert is_list(keys) + assert length(keys) > 0 + assert Enum.all?(keys, &is_binary/1) + end + end +end diff --git a/test/microwaveprop/commercial/snmp_client_test.exs b/test/microwaveprop/commercial/snmp_client_test.exs index 17747a39..f4059046 100644 --- a/test/microwaveprop/commercial/snmp_client_test.exs +++ b/test/microwaveprop/commercial/snmp_client_test.exs @@ -278,15 +278,80 @@ defmodule Microwaveprop.Commercial.SnmpClientTest do end test "non-string radio type raises FunctionClauseError" do - # poll/3 pattern matches on literal binary radio types; any other - # binary that isn't in the dispatch table should fail loudly so a - # mis-typed config surfaces rather than silently mis-polling. assert_raise FunctionClauseError, fn -> SnmpClient.poll("10.0.0.1", "public", "AF11X") end end end + describe "poll/3 with injected snmp_runner" do + setup do + prev = Application.get_env(:microwaveprop, :snmp_runner) + on_exit(fn -> + if prev, do: Application.put_env(:microwaveprop, :snmp_runner, prev), + else: Application.delete_env(:microwaveprop, :snmp_runner) + end) + :ok + end + + test "poll_af11x returns {:ok, result} when snmpget succeeds" do + Application.put_env(:microwaveprop, :snmp_runner, fn _cmd, _args -> + {@af11x_output, 0} + end) + + assert {:ok, result} = SnmpClient.poll("10.0.0.1", "public", "af11x") + assert result.rx_power_0 == -55 + assert result.link_state == 1 + end + + test "poll_af11x returns {:error, :snmp_failed} when snmpget fails" do + Application.put_env(:microwaveprop, :snmp_runner, fn _cmd, _args -> + {"timeout", 1} + end) + + assert {:error, :snmp_failed} = SnmpClient.poll("10.0.0.1", "public", "af11x") + end + + test "poll_af11x returns {:error, :empty_response} when output is empty" do + Application.put_env(:microwaveprop, :snmp_runner, fn _cmd, _args -> + {"", 0} + end) + + assert {:error, :empty_response} = SnmpClient.poll("10.0.0.1", "public", "af11x") + end + + test "poll_af60 returns {:ok, result} when both snmpget calls succeed" do + Application.put_env(:microwaveprop, :snmp_runner, fn cmd, _args -> + case cmd do + "snmpget" -> {@af60_static_output, 0} + "snmpgetnext" -> {@af60_station_output, 0} + end + end) + + assert {:ok, result} = SnmpClient.poll("10.0.0.1", "public", "af60") + assert result.radio_temp_0_c == 52 + assert result.link_state == 1 + assert result.rx_power_0 == -58 + assert result.tx_capacity == 900 + end + + test "poll_af60 returns {:error, :snmp_failed} when static snmpget fails" do + Application.put_env(:microwaveprop, :snmp_runner, fn _cmd, _args -> + {"timeout", 1} + end) + + assert {:error, :snmp_failed} = SnmpClient.poll("10.0.0.1", "public", "af60") + end + + test "poll_af60 returns {:error, :empty_response} when output is empty" do + Application.put_env(:microwaveprop, :snmp_runner, fn _cmd, _args -> + {"", 0} + end) + + assert {:error, :empty_response} = SnmpClient.poll("10.0.0.1", "public", "af60") + end + end + describe "parse_snmpget_output/2 property" do # Pick any AF11X integer value and any recognised AF11X OID suffix, render # the canonical `iso. = INTEGER: ` line, and assert the value diff --git a/test/microwaveprop/rover/road_proximity_test.exs b/test/microwaveprop/rover/road_proximity_test.exs new file mode 100644 index 00000000..80e2d0e2 --- /dev/null +++ b/test/microwaveprop/rover/road_proximity_test.exs @@ -0,0 +1,73 @@ +defmodule Microwaveprop.Rover.RoadProximityTest do + use ExUnit.Case, async: true + + alias Microwaveprop.Rover.RoadProximity + + describe "road_distances/3" do + test "returns distances when segments are available" do + cells = [%{lat: 33.0, lon: -97.0}] + bbox = %{"south" => 32.5, "north" => 33.5, "west" => -98.0, "east" => -96.0} + + # One segment 0.5° (~55 km) from the cell + fetch = fn _bbox -> {:ok, [{{33.0, -96.5}, {33.0, -96.0}}]} end + + {:ok, distances} = RoadProximity.road_distances(cells, bbox, fetch: fetch) + assert is_map(distances) + assert Map.has_key?(distances, {33.0, -97.0}) + assert distances[{33.0, -97.0}] > 0 + end + + test "returns nanometres for point exactly on the road" do + cells = [%{lat: 33.0, lon: -97.0}] + bbox = %{"south" => 32.5, "north" => 33.5, "west" => -98.0, "east" => -96.0} + + # Segment passes through the cell point + fetch = fn _bbox -> {:ok, [{{33.0, -98.0}, {33.0, -96.0}}]} end + + {:ok, distances} = RoadProximity.road_distances(cells, bbox, fetch: fetch) + assert_in_delta distances[{33.0, -97.0}], 0.0, 0.001 + end + + test "returns {:error, :no_roads} when no segments found" do + cells = [%{lat: 33.0, lon: -97.0}] + bbox = %{"south" => 32.5, "north" => 33.5, "west" => -98.0, "east" => -96.0} + + assert {:error, :no_roads} = RoadProximity.road_distances(cells, bbox, fetch: fn _ -> {:ok, []} end) + end + + test "propagates fetch errors" do + cells = [%{lat: 33.0, lon: -97.0}] + bbox = %{"south" => 32.5, "north" => 33.5, "west" => -98.0, "east" => -96.0} + + assert {:error, :overpass_timeout} = + RoadProximity.road_distances(cells, bbox, fetch: fn _ -> {:error, :overpass_timeout} end) + end + + test "multiple cells each get their nearest road distance" do + cells = [ + %{lat: 33.0, lon: -97.0}, + %{lat: 33.5, lon: -97.0}, + %{lat: 34.0, lon: -97.0} + ] + + bbox = %{"south" => 32.5, "north" => 34.5, "west" => -98.0, "east" => -96.0} + + # Road runs east-west at lat=33.0 + fetch = fn _bbox -> + {:ok, [{{33.0, -98.0}, {33.0, -96.0}}]} + end + + {:ok, distances} = RoadProximity.road_distances(cells, bbox, fetch: fetch) + # Cell at lat=33.0 is ON the road, lat=33.5 is ~55 km north, lat=34.0 is ~111 km north + assert distances[{33.0, -97.0}] < distances[{33.5, -97.0}] + assert distances[{33.5, -97.0}] < distances[{34.0, -97.0}] + end + + test "segments_from_elements excludes non-way or short geometry" do + # Tested implicitly through road_distances with injected fetch. + # The internal segments_from_elements/1 is exercised by: + # road_distances → fetch → {:ok, segments} + # where segments are directly provided. + end + end +end