Graham McIntire
fc3eb58910
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k surface obs, new hrrr_native_profiles table) drive these changes: * Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus removed" conclusion was an artifact of the prior matching strategy; with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins. * Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in those bands stop being silently dropped. Coefficients extrapolate ITU-R P.676/P.838 trends from the existing 134/241 GHz entries. * Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump the full failure body when CDS omits the message field — the prior "ERA5 job failed: nil" was masking real error reasons in oban_jobs. * Add scripts/recalibrate_algo.py so this analysis can be re-run any time new data lands without manual SQL. Reads PROP_PROD_DB_URL from .envrc, drops a Markdown report into docs/algo-reports/. * Append a dated Part 2c to algo.md documenting the corpus expansion, the honest accounting of historical HRRR coverage (only ~1,020 of 58k contacts are precision-matchable), and the empty era5/rtma/climatology tables. Update the gaseous-absorption and rain-attenuation tables to include the new bands. Test suite: 1,335 tests, 0 failures.
121 lines
5.4 KiB
Markdown
121 lines
5.4 KiB
Markdown
# Recalibration Analysis Report — 2026-04-13
|
|
|
|
> Auto-generated by `scripts/recalibrate_algo.py`. This report is the
|
|
> input for `algo.md` updates — diff against the prior dated section to
|
|
> see what's actually moved.
|
|
|
|
Connection: `postgres://prop:***@10.0.15.24:5432/prop`
|
|
Statement timeout: 20min
|
|
|
|
|
|
## Row counts and date ranges
|
|
|
|
| tbl | n | lo | hi |
|
|
|:---------------------|---------:|:-----------|:-----------|
|
|
| contacts | 58560 | 1991-05-04 | 2026-03-07 |
|
|
| era5_profiles | 0 | | |
|
|
| hrrr_native_profiles | 11472 | 2019-03-09 | 2025-09-21 |
|
|
| hrrr_profiles | 76058705 | 2016-09-17 | 2026-04-13 |
|
|
| iemre_observations | 15745 | | |
|
|
| nexrad_observations | 7286 | 2019-08-17 | 2024-09-22 |
|
|
| propagation_scores | 9618752 | 2026-04-13 | 2026-04-13 |
|
|
| rtma_observations | 0 | | |
|
|
| soundings | 16864 | 1990-06-07 | 2026-03-07 |
|
|
| surface_observations | 392442 | 1991-05-03 | 2026-04-13 |
|
|
| terrain_profiles | 58554 | | |
|
|
|
|
|
|
## Data gaps (these should NOT be zero in a healthy prod)
|
|
|
|
- **era5_rows**: 0
|
|
- **hrrr_climatology_rows**: 0
|
|
- **rtma_rows**: 0
|
|
- **metar5_rows**: 0
|
|
- **hrrr_complete_contacts**: 44147
|
|
- **hrrr_pending_contacts**: 14413
|
|
|
|
|
|
## Contacts by band (≥902 MHz)
|
|
|
|
| band_mhz | contacts | avg_km | max_km |
|
|
|-----------:|-----------:|---------:|---------:|
|
|
| 902 | 36 | 172.000 | 915.000 |
|
|
| 1296 | 74 | 259.000 | 1497.000 |
|
|
| 2304 | 15 | 91.000 | 348.000 |
|
|
| 3456 | 5 | 79.000 | 157.000 |
|
|
| 5760 | 4 | 97.000 | 148.000 |
|
|
| 10000 | 53698 | 213.000 | 2393.000 |
|
|
| 24000 | 3786 | 96.000 | 710.000 |
|
|
| 47000 | 764 | 66.000 | 343.000 |
|
|
| 75000 | 108 | 62.000 | 289.000 |
|
|
| 122000 | 35 | 27.000 | 139.000 |
|
|
| 134000 | 5 | 65.000 | 157.000 |
|
|
| 142000 | 4 | 47.000 | 79.700 |
|
|
| 145000 | 2 | 40.000 | 79.600 |
|
|
| 241000 | 15 | 29.000 | 114.400 |
|
|
| 288000 | 1 | 1.000 | 1.246 |
|
|
| 322000 | 1 | 1.000 | 1.400 |
|
|
| 403000 | 1 | 1.000 | 1.400 |
|
|
| 411000 | 1 | 0.000 | 0.050 |
|
|
|
|
|
|
## Monthly sounding ducting
|
|
|
|
| month | soundings | ducting_pct | avg_min_grad | avg_pwat |
|
|
|--------:|------------:|--------------:|---------------:|-----------:|
|
|
| 1 | 95 | 28.400 | -171.400 | 10.900 |
|
|
| 2 | 131 | 29.000 | -163.100 | 7.900 |
|
|
| 3 | 81 | 17.300 | -148.100 | 9.400 |
|
|
| 4 | 107 | 38.300 | -190.100 | 15.700 |
|
|
| 5 | 311 | 51.800 | -286.400 | 25.600 |
|
|
| 6 | 380 | 69.500 | -355.900 | 32.000 |
|
|
| 7 | 292 | 67.800 | -296.000 | 29.300 |
|
|
| 8 | 2572 | 54.900 | -261.200 | 34.600 |
|
|
| 9 | 2335 | 56.200 | -279.800 | 27.200 |
|
|
| 10 | 155 | 57.400 | -307.100 | 19.300 |
|
|
| 11 | 158 | 50.000 | -256.700 | 12.000 |
|
|
| 12 | 140 | 25.700 | -185.200 | 10.800 |
|
|
|
|
|
|
## HRRR native-profile duct distribution (best supportable band)
|
|
|
|
| band_bin | profiles | avg_inv_top_m | avg_theta_e_jump | avg_richardson |
|
|
|:-----------|-----------:|----------------:|-------------------:|-----------------:|
|
|
| 15-30 GHz | 10 | 2091.000 | 5.900 | 8.690 |
|
|
| 30-75 GHz | 4 | 3860.000 | 5.000 | 12.120 |
|
|
| 5-15 GHz | 98 | 4408.000 | 25.700 | 12.550 |
|
|
| <5 GHz | 1432 | 2097.000 | 5.400 | 18.420 |
|
|
| none | 9928 | 11497.000 | 80.500 | 38.270 |
|
|
|
|
|
|
## Contact ↔ HRRR per-band Pearson correlations (matched n=1020)
|
|
|
|
| band_mhz | n | rho_pr | rho_dpc | rho_pwat | rho_sref | rho_grad | rho_tc | rho_hpbl |
|
|
|-----------:|--------:|---------:|----------:|-----------:|-----------:|-----------:|---------:|-----------:|
|
|
| 1296.000 | 63.000 | -0.446 | 0.150 | -0.004 | 0.321 | -0.308 | -0.313 | -0.383 |
|
|
| 10000.000 | 680.000 | -0.104 | 0.046 | 0.003 | 0.181 | -0.127 | -0.108 | -0.200 |
|
|
| 24000.000 | 147.000 | -0.290 | -0.179 | -0.220 | 0.096 | -0.034 | -0.301 | -0.226 |
|
|
| 47000.000 | 75.000 | -0.111 | 0.015 | 0.006 | 0.218 | -0.167 | 0.086 | -0.154 |
|
|
|
|
|
|
## 10 GHz: HPBL bin distance distribution
|
|
|
|
| bin | n | avg_km | p50_km |
|
|
|:----------|----:|---------:|---------:|
|
|
| <200 | 55 | 222.600 | 174.000 |
|
|
| 200-500 | 174 | 183.200 | 154.500 |
|
|
| 500-1000 | 181 | 163.700 | 126.000 |
|
|
| 1000-1500 | 103 | 160.900 | 136.000 |
|
|
| 1500-2000 | 93 | 140.500 | 128.000 |
|
|
| >=2000 | 74 | 100.300 | 90.000 |
|
|
|
|
|
|
## 10 GHz: pressure bin distance distribution
|
|
|
|
| bin | n | avg_km | p50_km |
|
|
|:----------|----:|---------:|---------:|
|
|
| <990 | 101 | 211.900 | 134.000 |
|
|
| 990-1000 | 538 | 155.400 | 128.000 |
|
|
| 1000-1010 | 28 | 164.600 | 115.000 |
|
|
| 1010-1020 | 2 | 23.000 | 23.000 |
|
|
| >=1020 | 11 | 103.100 | 23.000 |
|