Three signal sources we already collect but weren't using:
* NEXRAD composite reflectivity → rain rate via Marshall-Palmer, taken
as max of HRRR-derived and NEXRAD-derived rate so fast convective
cells between HRRR hourly analyses can still trigger the rain penalty.
Only active on f00 — forecast hours can't see future radar. New
Scorer.dbz_to_rain_rate_mmhr/1 with 5 dBZ noise floor and 150 mm/hr
hail-safe ceiling.
* hrrr_native_profiles.best_duct_band_ghz → Scorer.score_refractivity/4
applies a 1.15× boost when the cell's native-resolution duct supports
the target band's frequency. HRRR pressure-level gradients
systematically under-read thin trapping layers the native profile can
resolve. Sub-band ducts do NOT boost — they're evidence that the
gradient we have is all there is at the target frequency.
* Commercial LOS link rx_power fading → inverse tropo sensor.
Commercial.link_degradation_at/3 computes the average 7-day-baseline
vs current delta across enabled links within 75 km, ignoring links
where link_state != 1. Scorer.commercial_link_boost/2 adds +2 to +25
to the composite score for 3+ dB of fading. ~150 km radius around
DFW is the only zone this helps today, but it's the first *measured*
signal in the algorithm vs the model-derived proxies.
Also fix a latent test bug exposed by the earlier ERA5 poll-timeout
bump: era5_batch_client_test's "uncached path returns error" tests
hung for up to an hour when run with direnv's real CDS key. New
describe-level setup explicitly unsets the env var so the tests stay
hermetic.
1,359 tests, 0 failures.
Direct queries against prod (75M HRRR profiles, 16,864 soundings, 392k
surface obs, new hrrr_native_profiles table) drive these changes:
* Reinstate shallow-BL bonus as an HPBL multiplier on the refractivity
score (1.10× <200m → 0.78× ≥2000m). The Apr 11 "shallow BL bonus
removed" conclusion was an artifact of the prior matching strategy;
with the cleaner contact↔HRRR join (n=680) the binned data goes 230 km
avg at HPBL <200m vs 100 km at ≥2000m, monotonic across all bins.
* Add 6 missing bands (142, 145, 288, 322, 403, 411 GHz) so contacts in
those bands stop being silently dropped. Coefficients extrapolate
ITU-R P.676/P.838 trends from the existing 134/241 GHz entries.
* Bump ERA5 poll timeout 10 min → 1 hour and Era5MonthBatchWorker
max_attempts 3 → 5 so CDS slowness stops discarding tiles. Also dump
the full failure body when CDS omits the message field — the prior
"ERA5 job failed: nil" was masking real error reasons in oban_jobs.
* Add scripts/recalibrate_algo.py so this analysis can be re-run any
time new data lands without manual SQL. Reads PROP_PROD_DB_URL from
.envrc, drops a Markdown report into docs/algo-reports/.
* Append a dated Part 2c to algo.md documenting the corpus expansion,
the honest accounting of historical HRRR coverage (only ~1,020 of 58k
contacts are precision-matchable), and the empty era5/rtma/climatology
tables. Update the gaseous-absorption and rain-attenuation tables to
include the new bands.
Test suite: 1,335 tests, 0 failures.
- Replace stale April 2026 manual weights with recalibrated values
- Document native hybrid-sigma data in data flow section
- Note refractivity factor now uses native 10-50m resolution
- Add hourly grid integration section to Part 12
- Expose duct_info (count, freq, thickness) in scoring factors for UI
Backtest on 11,431 native profiles (2026-04-11):
- Drop duct_usable_10/24/47ghz (always 1.0, no discrimination)
- Drop bulk_richardson (near-identical QSO vs baseline means)
- Document all feature results with signal strength assessment
- theta_e_jump is strongest native discriminator (44% lift)
- best_duct_freq and duct_thickness show clear physical signal
Add sections for all three new data sources in Meteorological
Foundations: path-integrated scoring strategy, ERA5 reanalysis
for pre-2014 contacts, RTMA 15-minute surface analysis. Update
calibration dataset summary with ERA5/RAOB/RTMA coverage.
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
Comprehensive description of atmospheric science behind the predictions
for meteorologist review: refractivity-propagation connection, four
ducting mechanisms, HRRR model usage, frequency-dependent effects,
diurnal/seasonal cycles, and known limitations.
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
- Replace piecewise knife-edge loss with P.526-16 Eq. 31 single formula
- Fix diffraction parameter ν to use standard formula instead of ad-hoc approximation
- Implement Deygout 3-edge method for multiple obstacle diffraction
- Add dynamic k-factor from HRRR refractivity gradient (falls back to 4/3)
- Terrain worker now looks up nearest HRRR profile for atmospheric correction
- Update algo.md with P.526-16 methods and k-factor table
- Fix pre-existing map_live_test antenna height default (33 ft, not 8)
- Expand ITU-R section with recommendation numbers, titles, and usage
- Document all 8 external data sources with API URLs, volumes, fields
- Add derived parameter documentation (sounding params, HRRR profiles)
- Document live scoring pipeline (grid resolution, update cadence, retention)
- Add contest bias and rain model caveats to known issues
- Document Finding 11: regional QSO performance by Maidenhead field
- Explain why regional weight adjustments are not justified
- Fix algo.md table formatting (blank lines before tables for Earmark)
- Change page title from Phoenix Framework to North Texas Microwave Society
Poll UBNT AirFiber radios (AF11X + AF60-LR) every 5 minutes via
net-snmp CLI, storing signal metrics in commercial_samples. Fetches
ASOS weather alongside each cycle for propagation correlation.
Includes 7 seeded link definitions, Oban cron worker, and net-snmp
in the Docker image.
