Commit graph

23 commits

Author SHA1 Message Date
880988b591 Update algo.md weights and data flow for native duct integration
- Replace stale April 2026 manual weights with recalibrated values
- Document native hybrid-sigma data in data flow section
- Note refractivity factor now uses native 10-50m resolution
- Add hourly grid integration section to Part 12
- Expose duct_info (count, freq, thickness) in scoring factors for UI
2026-04-11 13:44:13 -05:00
7d68d13dcc Recalibrate scoring weights from gradient descent on 5000 QSOs
Loss improved 72% (0.42 → 0.12). Key changes:
- rain: 0.08 → 0.136 (+70%) — strongest discriminator
- season: 0.08 → 0.111 (+39%)
- wind: 0.05 → 0.08 (+60%)
- refractivity: 0.08 → 0.105 (+31%)
- time_of_day: 0.10 → 0.050 (-50%) — was overweighted by contest bias
- pressure: 0.15 → 0.103 (-31%)
- humidity: 0.18 → 0.124 (-31%)

Validated by native profile backtest (11,431 profiles):
theta_e_jump strongest native discriminator, duct_usable_* and
bulk_richardson dropped as dead features.
2026-04-11 13:20:00 -05:00
d49ec5d32e Drop dead backtest features, document consolidated results
Backtest on 11,431 native profiles (2026-04-11):
- Drop duct_usable_10/24/47ghz (always 1.0, no discrimination)
- Drop bulk_richardson (near-identical QSO vs baseline means)
- Document all feature results with signal strength assessment
- theta_e_jump is strongest native discriminator (44% lift)
- best_duct_freq and duct_thickness show clear physical signal
2026-04-11 13:11:39 -05:00
c8127fbb02 Update algo.md to match current codebase
- Fix pressure scoring thresholds (finer granularity at <980-990 mb)
- Fix 10 GHz parameters (o2_db_km, h2o_coeff, seasonal Feb/Oct)
- Fix 24 GHz parameters (o2_db_km 0.02, h2o_coeff 0.002)
- Fix 47 GHz o2_db_km (0.04)
- Add 5 new UHF/microwave bands (902-5760 MHz)
- Add Part 12: HRRR native hybrid-sigma levels (50-level profiles)
- Add Part 13: Duct analysis (M-profile, trapped frequency)
- Add Part 14: Inversion analysis (Ri, theta-e, shear)
- Add Part 15: NEXRAD radar data integration
- Add Part 16: Backtest framework (21 features)
2026-04-11 10:19:08 -05:00
0977d041b8 Document ERA5, RTMA, and path-integrated HRRR in algo.md
Add sections for all three new data sources in Meteorological
Foundations: path-integrated scoring strategy, ERA5 reanalysis
for pre-2014 contacts, RTMA 15-minute surface analysis. Update
calibration dataset summary with ERA5/RAOB/RTMA coverage.
2026-04-07 12:37:51 -05:00
2eefe64c9b Update seasonal scores from sounding data, document commercial link patterns
Soundings (10,935): October has strongest gradients (-350 N/km) and
65.7% ducting, raised seasonal score 82→88. February ducting (20.2%)
exceeds January (14.8%), raised 32→40.

Commercial links (12,068 samples): 68 GHz shows 3.9 dB diurnal swing,
11 GHz shows 1.7 dB with inverted multipath pattern, 24 GHz remarkably
stable at 0.9 dB. Diurnal sensitivity is non-monotonic with frequency.

RAOB gradients avg -265 N/km vs HRRR -107 (2.5x stronger) — confirms
HRRR misses thin surface ducts that soundings resolve.
2026-04-07 11:45:44 -05:00
3cdd132f24 Data-driven algorithm refinements from full dataset analysis
Analyzed 37,925 HRRR-matched contacts from 58,367 total.

- Remove shallow BL bonus (score 82 for HPBL < 300m): data shows
  medium BL (1000-2000m) produces longest contacts (222 km avg),
  not shallow (210 km). Refractivity fallback now uses default score.

- Refine pressure scoring bins: add <980 mb tier (score 88), steeper
  gradient from low to high. Contacts at <970 mb avg 242.7 km vs
  184.1 km at 990-1000 mb.

- Update algo.md calibration stats (58,367 contacts, 41M HRRR
  profiles, 58,361 terrain profiles) and document dataset bias
  (99.5% Aug-Sep).

- Create updates.md with full binned analysis tables for all factors.
2026-04-07 11:15:49 -05:00
4af6c5db73
Fix stale pressure trend scores and pressure levels in algo.md 2026-04-01 18:28:10 -05:00
7fa8ca1ebe
Remove reference to 'original model' in algo.md 2026-04-01 16:52:46 -05:00
dd8ac4a311
Rewrite meteorological foundations for technical audience 2026-04-01 16:21:21 -05:00
bc529af392
Update algo.md Finding 10: corrected mode advantage, no SSB on rainscatter 2026-04-01 11:10:11 -05:00
5c9b43d619
Update analysis with solar time, refine algo.md time-of-day section
Solar time (longitude/15) replaces fixed CDT/CST offset for time-of-day
scoring. Correlation analysis shows dramatic improvement at higher
frequencies: 24 GHz rho jumps from 0.056 (UTC) to 0.188 (solar), and
75 GHz corrects from spurious -0.39 to physically correct +0.24.
2026-04-01 09:02:57 -05:00
83a1511c84
Update algo.md with data-driven findings from 57K QSO analysis 2026-04-01 08:41:33 -05:00
88655e40a6
Add meteorological foundations section to algo.md
Comprehensive description of atmospheric science behind the predictions
for meteorologist review: refractivity-propagation connection, four
ducting mechanisms, HRRR model usage, frequency-dependent effects,
diurnal/seasonal cycles, and known limitations.
2026-03-31 17:00:30 -05:00
c9112b9280
Recalibrate refractivity thresholds for HRRR gradient distribution
Previous thresholds (-500 to -60) were calibrated for radiosonde data.
HRRR profiles have coarser vertical resolution, with gradients clustering
between -40 and -130 N/km (median -70). Nearly all grid points were
falling through to the default score of 42, wasting the refractivity
factor. New thresholds (-200 to -40) spread across HRRR percentiles.
2026-03-31 16:08:15 -05:00
49cbe6789c
Implement ITU-R P.526-16 terrain diffraction model
- Replace piecewise knife-edge loss with P.526-16 Eq. 31 single formula
- Fix diffraction parameter ν to use standard formula instead of ad-hoc approximation
- Implement Deygout 3-edge method for multiple obstacle diffraction
- Add dynamic k-factor from HRRR refractivity gradient (falls back to 4/3)
- Terrain worker now looks up nearest HRRR profile for atmospheric correction
- Update algo.md with P.526-16 methods and k-factor table
- Fix pre-existing map_live_test antenna height default (33 ft, not 8)
2026-03-31 16:04:23 -05:00
d458c23782
Expand algo.md sources: full data provenance, APIs, and ITU-R references
- Expand ITU-R section with recommendation numbers, titles, and usage
- Document all 8 external data sources with API URLs, volumes, fields
- Add derived parameter documentation (sounding params, HRRR profiles)
- Document live scoring pipeline (grid resolution, update cadence, retention)
- Add contest bias and rain model caveats to known issues
2026-03-31 10:52:45 -05:00
a18cf48493
Add regional analysis to algo.md, update site title to NTMS
- Document Finding 11: regional QSO performance by Maidenhead field
- Explain why regional weight adjustments are not justified
- Fix algo.md table formatting (blank lines before tables for Earmark)
- Change page title from Phoenix Framework to North Texas Microwave Society
2026-03-31 10:47:39 -05:00
a6b4ac1965
Increase weather queue to 20 and iemre to 10 for faster backfill 2026-03-31 10:41:49 -05:00
d7fc120c28
Update algo link data location and fix markdown dark mode styling 2026-03-31 09:39:12 -05:00
ea439e56bd
bump algo and fix backfill worker 2026-03-30 15:54:04 -05:00
10e8feb486
Add commercial link monitoring via SNMP polling
Poll UBNT AirFiber radios (AF11X + AF60-LR) every 5 minutes via
net-snmp CLI, storing signal metrics in commercial_samples. Fetches
ASOS weather alongside each cycle for propagation correlation.

Includes 7 seeded link definitions, Oban cron worker, and net-snmp
in the Docker image.
2026-03-30 13:02:59 -05:00
42b2ac920b
add algo docs 2026-03-29 15:55:34 -05:00