feat(propagation): per-band weight calibration from full-corpus correlation

Ran recalibrate_algo.py against the full local prop_dev (81,994 contacts,
18.6M HRRR rows) and derived per-band composite weights for the nine
bands with >=200 matched contacts. Moisture (dewpoint/PWAT/surface N)
is consistently beneficial through 5.76 GHz, reverses at 24 GHz; rain
scales sqrt(rain_k) not linearly so 24 GHz gets 0.215 rain weight
instead of the linear-ratio 0.95. 10 GHz stays as the reference band
(defaults); 47+ GHz inherits defaults (n<200).

- BandConfig.weights/1 returns per-band override or default fallback
- @band_configs carries :weights on 222/432/902/1296/2304/3400/5760/24G
- Recalibrator.compute_factors/3 + fit(band_mhz:) band-aware fitting
- Scorer + ContactLive.Show pass band_config to weights/1
- algo.md Part 2d documents the 2026-04-18 analysis + derivation rule
- scripts/derive_band_weights.py turns correlations into weight maps
- report preserved at docs/algo-reports/2026-04-18-recalibration.md

2026-04-18 10:19:26 -05:00

16 KiB

Raw Blame History

Recalibration Analysis Report — 2026-04-18

Auto-generated by scripts/recalibrate_algo.py. This report is the input for algo.md updates — diff against the prior dated section to see what's actually moved.

Connection: postgresql://postgres@localhost:5432/prop_dev Statement timeout: 60min

Row counts and date ranges

tbl	n	lo	hi
contacts	81994.000	1991-05-04	2026-03-07
hrrr_native_profiles	11472.000	2019-03-09	2025-09-21
hrrr_profiles	18591740.000	2018-08-04	2026-04-05
iemre_observations	28438.000
narr_profiles	354.000	1991-05-04	2014-09-19
nexrad_observations	7286.000	2019-08-17	2024-09-22
propagation_scores	nan
rtma_observations	0.000
soundings	26933.000	1990-06-07	2026-04-18
surface_observations	599046.000	1991-05-03	2026-04-18
terrain_profiles	81994.000

Data gaps (these should NOT be zero in a healthy prod)

narr_rows: 354
narr_pre2014_rows: 354
hrrr_climatology_rows: 0
rtma_rows: 0
metar5_rows: 0
hrrr_complete_contacts: 67260
hrrr_pending_contacts: 14734
pre2014_contacts: 207

Contacts by band (≥50 MHz)

band_mhz	contacts	avg_km	max_km
222	7595	229.000	2306.000
432	9177	200.000	1752.000
902	1794	171.000	915.000
1296	2996	160.000	1752.000
2304	749	153.000	655.000
3400	351	151.000	643.000
5760	328	125.000	505.000
10000	54264	212.000	2393.000
24000	3806	95.000	710.000
47000	762	65.000	343.000
75000	106	60.000	289.000
122000	35	27.000	139.000
134000	5	65.000	157.000
142000	4	47.000	79.700
145000	2	40.000	79.600
241000	14	31.000	114.400
288000	1	1.000	1.246
322000	1	1.000	1.400
403000	1	1.000	1.400
411000	1	0.000	0.050

Monthly sounding ducting

month	soundings	ducting_pct	avg_min_grad	avg_pwat
1	95	28.400	-171.400	10.900
2	131	29.000	-163.100	7.900
3	81	17.300	-148.100	9.400
4	134	31.300	-175.600	13.500
5	311	51.800	-286.400	25.600
6	380	69.500	-355.900	32.000
7	302	67.500	-293.800	29.700
8	5007	55.200	-255.400	35.200
9	2335	56.200	-279.800	27.200
10	155	57.400	-307.100	19.300
11	158	50.000	-256.700	12.000
12	140	25.700	-185.200	10.800

HRRR native-profile duct distribution (best supportable band)

band_bin	profiles	avg_inv_top_m	avg_theta_e_jump	avg_richardson
<5 GHz	1432	2097.000	5.400	18.420
15-30 GHz	10	2091.000	5.900	8.690
30-75 GHz	4	3860.000	5.000	12.120
5-15 GHz	98	4408.000	25.700	12.550
none	9928	11497.000	80.500	38.270

Contact ↔ HRRR per-band Pearson correlations (matched n=23885)

band_mhz	n	rho_pr	rho_dpc	rho_pwat	rho_sref	rho_grad	rho_tc	rho_hpbl
222.000	5392.000	-0.038	0.063	0.077	0.077	0.057	-0.086	-0.033
432.000	6583.000	0.058	0.137	0.124	0.136	0.041	-0.089	-0.066
902.000	1317.000	0.044	0.096	0.049	0.079	-0.014	-0.060	-0.010
1296.000	2146.000	0.013	0.085	0.049	0.085	-0.031	-0.032	-0.041
2304.000	564.000	-0.006	0.130	0.086	0.154	-0.110	-0.114	-0.074
3400.000	280.000	-0.025	0.150	0.131	0.176	0.046	-0.103	-0.023
5760.000	246.000	0.040	0.087	0.137	0.105	0.021	-0.082	-0.083
10000.000	6675.000	-0.066	-0.028	0.017	0.031	-0.024	0.043	-0.025
24000.000	613.000	-0.178	-0.208	-0.172	-0.103	-0.251	-0.007	-0.201
47000.000	53.000	-0.423	-0.518	-0.149	0.510	-0.154	0.222	-0.465

Per-band HPBL bin distance distribution

One table per band with ≥50 matched contacts. A band-specific effect here means HPBL belongs in the scorer for that band.

222 MHz:

bin	n	avg_km	p50_km
<200	956	273.500	210.000
200-500	1131	215.500	164.000
500-1000	1185	205.400	152.000
1000-1500	948	220.600	156.000
1500-2000	913	223.000	171.000
>=2000	259	246.900	192.000

432 MHz:

bin	n	avg_km	p50_km
<200	1258	223.000	179.000
200-500	1370	198.600	143.500
500-1000	1413	183.700	131.000
1000-1500	1124	198.000	145.000
1500-2000	1129	187.100	142.000
>=2000	289	169.200	126.000

902 MHz:

bin	n	avg_km	p50_km
<200	215	205.300	165.000
200-500	275	152.500	133.000
500-1000	302	166.500	123.000
1000-1500	259	175.300	137.000
1500-2000	222	175.900	152.000
>=2000	44	158.700	131.000

1.296 GHz:

bin	n	avg_km	p50_km
<200	355	179.600	151.000
200-500	484	159.300	129.500
500-1000	479	145.600	107.000
1000-1500	376	154.800	131.000
1500-2000	380	153.600	122.000
>=2000	72	143.700	108.000

2.304 GHz:

bin	n	avg_km	p50_km
<200	72	188.000	160.000
200-500	103	174.800	156.000
500-1000	155	150.300	122.000
1000-1500	121	151.600	136.000
1500-2000	97	161.000	147.000
>=2000	16	144.800	140.500

3.4 GHz:

bin	n	avg_km	p50_km
<200	39	157.400	131.000
200-500	57	162.500	140.000
500-1000	64	153.500	111.000
1000-1500	67	150.500	136.000
1500-2000	39	150.300	122.000
>=2000	14	157.400	140.500

5.76 GHz:

bin	n	avg_km	p50_km
<200	24	174.500	138.500
200-500	49	134.600	135.000
500-1000	62	118.000	103.000
1000-1500	54	125.000	133.000
1500-2000	45	114.600	122.000
>=2000	12	142.700	140.500

10 GHz:

bin	n	avg_km	p50_km
<200	1573	213.900	178.800
200-500	2146	201.300	175.000
500-1000	1663	211.800	158.100
1000-1500	742	214.500	212.500
1500-2000	410	179.400	173.000
>=2000	138	201.000	173.000

24 GHz:

bin	n	avg_km	p50_km
<200	115	120.600	110.000
200-500	236	95.400	99.000
500-1000	189	79.700	80.800
1000-1500	44	103.800	99.000
1500-2000	10	46.800	34.000
>=2000	19	69.200	23.200

47 GHz:

bin	n	avg_km	p50_km
<200	9	111.200	85.300
200-500	19	82.200	58.500
500-1000	16	55.600	27.600
1000-1500	5	16.100	11.400
>=2000	4	31.800	31.800

Per-band pressure bin distance distribution

Surface pressure tends to proxy synoptic-scale stagnation, so we want to see longer distances under the 1015-1025 mb ridge at every band that ducts.

222 MHz:

bin	n	avg_km	p50_km
<990	1967	228.800	177.000
990-1000	1289	248.500	191.000
1000-1010	1263	235.400	186.000
1010-1020	778	183.300	141.000
>=1020	95	159.300	126.000

432 MHz:

bin	n	avg_km	p50_km
<990	2548	189.600	140.000
990-1000	1580	223.200	173.000
1000-1010	1285	202.700	149.000
1010-1020	1050	172.700	129.000
>=1020	120	144.800	84.500

902 MHz:

bin	n	avg_km	p50_km
<990	422	157.600	131.500
990-1000	400	206.400	157.000
1000-1010	241	189.200	147.000
1010-1020	223	138.800	143.000
>=1020	31	72.000	32.000

1.296 GHz:

bin	n	avg_km	p50_km
<990	696	149.600	122.000
990-1000	579	183.700	145.000
1000-1010	451	161.600	108.000
1010-1020	385	130.500	114.000
>=1020	35	112.700	112.000

2.304 GHz:

bin	n	avg_km	p50_km
<990	138	155.100	136.000
990-1000	201	168.200	141.000
1000-1010	97	178.500	117.000
1010-1020	116	147.200	152.000
>=1020	12	126.300	137.000

3.4 GHz:

bin	n	avg_km	p50_km
<990	75	157.400	138.000
990-1000	134	156.800	135.000
1000-1010	34	179.700	131.000
1010-1020	28	122.200	124.000
>=1020	9	114.600	131.000

5.76 GHz:

bin	n	avg_km	p50_km
<990	55	112.700	111.000
990-1000	120	131.600	119.500
1000-1010	27	128.000	100.000
1010-1020	36	146.900	152.000
>=1020	8	123.000	135.000

10 GHz:

bin	n	avg_km	p50_km
<990	3515	222.700	196.600
990-1000	1620	189.700	152.700
1000-1010	803	194.200	153.000
1010-1020	651	182.900	133.900
>=1020	86	187.700	158.100

24 GHz:

bin	n	avg_km	p50_km
<990	282	97.300	98.700
990-1000	200	97.900	99.000
1000-1010	57	95.200	102.300
1010-1020	70	70.600	80.800
>=1020	4	103.600	103.600

47 GHz:

bin	n	avg_km	p50_km
<990	19	80.500	43.900
990-1000	25	62.600	31.800
1000-1010	2	84.600	84.600
1010-1020	7	56.400	58.500

Contact ↔ NARR per-band Pearson correlations (pre-2014 only, matched n=201)

Historical sanity check: if the HRRR-era signs and magnitudes survive on 30+ years of NARR reanalysis, the scoring factors are physical rather than HRRR-artefact. Diverging signs = flag it.

band_mhz	n	rho_pr	rho_dpc	rho_pwat	rho_sref	rho_grad	rho_tc	rho_hpbl
10000.000	103.000	-0.402	-0.402	-0.402	-0.220	-0.248	-0.402	-0.402

NEXRAD composite reflectivity vs distance (matched n=20900)

Rain-attenuation sanity check — one table per band with ≥50 matched contacts. At rain-sensitive bands (24+ GHz) higher max_dbz should map to shorter contacts; at 10 GHz and below the effect should be barely detectable.

10 GHz:

bin	n	avg_km	p50_km
none (<5)	14110	210.000	197.000
drizzle (5-20)	1284	200.900	199.300
light (20-30)	1736	211.800	197.100
moderate (30-45)	1609	230.900	215.200
heavy (>=45)	677	191.900	146.900

24 GHz:

bin	n	avg_km	p50_km
none (<5)	869	90.900	83.700
drizzle (5-20)	74	108.300	123.200
light (20-30)	151	88.200	70.900
moderate (30-45)	76	109.300	122.400
heavy (>=45)	49	86.600	102.300

47 GHz:

bin	n	avg_km	p50_km
none (<5)	186	52.400	56.000
light (20-30)	30	59.300	66.300
moderate (30-45)	9	75.100	81.600
heavy (>=45)	15	71.400	82.800

hrrr_native_profiles.best_duct_band_ghz vs distance (matched n=57669)

Validates the 1.15× boost in Scorer.score_refractivity/4: contacts where the native duct supports the target band should run longer than those where it does not. One table per band with ≥50 matched contacts.

10 GHz:

bin	n	avg_km	p50_km
none	45395	212.700	195.000
<5 GHz	7151	200.500	171.800
5-10 GHz	444	200.600	174.200
10-24 GHz	146	203.300	225.200
24-47 GHz	5	177.200	133.300
>=47 GHz	23	147.500	131.500

24 GHz:

bin	n	avg_km	p50_km
none	3188.000	95.000	92.600
<5 GHz	477.000	94.900	83.900
5-10 GHz	36.000	97.500	113.700
10-24 GHz	2.000	44.700	44.700
>=47 GHz	6.000	131.500	131.500

47 GHz:

bin	n	avg_km	p50_km
none	552.000	64.300	65.400
<5 GHz	131.000	56.800	61.800
5-10 GHz	6.000	98.100	113.700

75 GHz:

bin	n	avg_km	p50_km
none	76.000	54.800	36.700
<5 GHz	15.000	13.900	14.600

Commercial-link rx_power degradation vs contemporaneous DFW contacts (matched n=0)

No DFW-zone contacts fall inside the commercial_samples date window yet. Expected to stay empty until the Aug/Sep contest season produces contacts that overlap with live SNMP polling.

16 KiB Raw Blame History Unescape Escape

Recalibration Analysis Report — 2026-04-18

Row counts and date ranges

Data gaps (these should NOT be zero in a healthy prod)

Contacts by band (≥50 MHz)

Monthly sounding ducting

HRRR native-profile duct distribution (best supportable band)

Contact ↔ HRRR per-band Pearson correlations (matched n=23885)

Per-band HPBL bin distance distribution

Per-band pressure bin distance distribution

Contact ↔ NARR per-band Pearson correlations (pre-2014 only, matched n=201)

NEXRAD composite reflectivity vs distance (matched n=20900)

hrrr_native_profiles.best_duct_band_ghz vs distance (matched n=57669)

Commercial-link rx_power degradation vs contemporaneous DFW contacts (matched n=0)

16 KiB

Raw Blame History