diff --git a/Dockerfile b/Dockerfile index cae49fd5..87f9b787 100644 --- a/Dockerfile +++ b/Dockerfile @@ -71,7 +71,7 @@ RUN mix release FROM ${RUNNER_IMAGE} AS final RUN apt-get update \ - && apt-get install -y --no-install-recommends libstdc++6 openssl libncurses6 locales ca-certificates \ + && apt-get install -y --no-install-recommends libstdc++6 openssl libncurses6 locales ca-certificates snmp \ && rm -rf /var/lib/apt/lists/* # Set the locale diff --git a/algo.md b/algo.md index 4d423619..596ae0cf 100644 --- a/algo.md +++ b/algo.md @@ -1,348 +1,1115 @@ -# How the Microwave Propagation Algorithm Works +# Microwave Propagation Algorithm — Unified -This document explains how microwaveprop scores current band conditions and predicts propagation for ham radio frequencies from 10 GHz and up. It's written for humans, not computers. +## Overview + +Propagation scoring and prediction for microwave amateur radio bands (10-241 GHz), calibrated against 57,492 QSOs with distance data, validated against 18,540 signal samples from 7 commercial terrestrial links at 11/24/68 GHz, and grounded in ITU-R atmospheric models. + +The algorithm has two operating regimes: +1. **Beyond-LOS** — Ham radio paths (50-1000+ km) where atmospheric ducting and refraction are essential. This is the primary use case. +2. **LOS** — Known fixed links or short paths with clear Fresnel clearance where gaseous absorption is the dominant variable. + +The regime distinction matters because refractivity effects are *inverted* between the two: enhanced refraction extends beyond-LOS range but causes multipath fading on short LOS paths. + +### Calibration Dataset + +**QSO data:** 57,492 tropospheric contacts across 13 bands (ARRL Microwave Contest, 2019-2024), filtered to exclude EME/satellite. Weather-matched against 95 ASOS stations (surface) and 9 RAOB stations (soundings). + +**Link data:** 18,540 samples from 7 commercial links near DFW (March 14-29 2026), correlated with KTKI ASOS and HRRR model refractivity profiles. No rain during observation period. + +**Confirmed long-range contacts:** +- 47 GHz: 116.0 km (Nov 2025), 98.8 km (Jun 2024) +- 24 GHz: 542.1 km (Sep 2002, longest), 101.0 km (Nov 2025) +- 10 GHz: 1,609 km (longest in dataset) --- -## What We're Trying to Do +## Part 1: Atmospheric Physics -Every microwave band behaves differently depending on the weather. The algorithm watches real-time surface weather observations from ASOS stations, upper-air soundings from weather balloons, and (when available) forecast data — then produces two things: +### Absolute Humidity -1. **A current score (0-100) for each band** telling you how good conditions are right now -2. **A 6-hour forecast** showing whether conditions are getting better or worse +The single most important weather variable. Temperature and relative humidity are proxies; absolute humidity (g/m^3) directly determines gaseous absorption. -The score maps to a simple scale: +``` +rho = 217 * (RH/100) * e_s / T_kelvin -| Score | Label | What It Means | -|-------|-------|---------------| -| 80-100 | Excellent | Exceptional propagation. Work the bands now. | -| 65-79 | Good | Above-average conditions. Worth getting on the air. | -| 50-64 | Marginal | Normal conditions. Local and regional contacts possible. | -| 33-49 | Poor | Below average. Short-range contacts only. | -| 0-32 | Negligible | Band is essentially dead for anything beyond line-of-sight. | +e_s = 6.112 * exp(17.67 * T_c / (T_c + 243.5)) # Magnus formula (hPa) +``` + +### Surface Refractivity (ITU-R P.453-14) + +``` +N = 77.6 * P / T + 3.73e5 * e / T^2 + +P: pressure (hPa) +T: absolute temperature (K) +e: water vapor pressure (hPa) = 6.112 * exp(17.67 * Td_c / (Td_c + 243.5)) +``` + +N is a compound variable: both dry air (pressure/temperature) and moisture contribute. At 10 GHz, higher N increases beam bending (beneficial for beyond-LOS). At 24+ GHz, higher N usually means more moisture = more absorption (harmful), though the refractivity benefit partially offsets this. + +### Modified Refractivity (M-units) + +``` +M = N + 0.157 * (h_agl) + +h_agl: height above ground level (m) +``` + +Ducting occurs where dM/dh < 0 (M decreases with height). Duct strength = delta-M across the inversion layer. + +### Gaseous Absorption (ITU-R P.676-13) + +Total absorption per km = O2 component (fixed) + H2O component (humidity-dependent): + +| Band | f (GHz) | O2 (dB/km) | H2O Coeff (dB/km per g/m^3) | Total @ 7.5 g/m^3 | Dominant Constraint | +|------|---------|------------|---------------------------|-------------------|---------------------| +| 10G | 10.368 | 0.008 | 0.0005 | 0.012 | Negligible absorption | +| 24G | 24.192 | 0.015 | 0.012 | 0.105 | 22.235 GHz H2O line | +| 47G | 47.088 | 0.045 | 0.003 | 0.068 | O2 wing + mild H2O | +| 68G | 68.040 | 0.90 | 0.007 | 0.95 | 60 GHz O2 band wing | +| 75G | 76.032 | 0.012 | 0.006 | 0.057 | Window band | +| 122G | 122.250 | 0.80 | 0.010 | 0.875 | 118.75 GHz O2 wing | +| 134G | 134.928 | 0.08 | 0.015 | 0.193 | Between O2 118 & H2O 183 | +| 142G | 142.000 | 0.05 | 0.025 | 0.238 | Approaching H2O 183 | +| 241G | 241.000 | 0.08 | 0.30 | 2.33 | Between H2O 183 & H2O 325 | + +The 11 GHz and 24 GHz coefficients are validated by commercial link measurements. The 68 GHz coefficient is directly measured (0.1 dB/km per g/m^3 increase on a 2.8 km path, consistent with ITU-R model when O2 wing contribution is included). + +### Rain Attenuation (ITU-R P.838-3) + +gamma_R = k * R^alpha (dB/km), R = rain rate (mm/hr): + +| Band | k_H | alpha_H | Light 4mm/hr | Moderate 10mm/hr | Heavy 25mm/hr | +|------|------|------|-------------|-----------------|--------------| +| 10G | 0.010 | 1.28 | 0.05 | 0.19 | 0.56 | +| 24G | 0.070 | 1.07 | 0.31 | 0.81 | 2.04 | +| 47G | 0.187 | 0.93 | 0.68 | 1.58 | 3.69 | +| 68G | 0.310 | 0.86 | 0.98 | 2.18 | 4.73 | +| 75G | 0.345 | 0.84 | 1.07 | 2.40 | 5.18 | +| 122G | 0.498 | 0.77 | 1.32 | 2.93 | 5.91 | +| 241G | 0.550 | 0.70 | 1.30 | 2.76 | 5.20 | + +Rain model is NOT validated by measured data (no rain events in link dataset). Coefficients are from ITU-R P.838-3 and interpolation. + +### Free-Space Path Loss (ITU-R P.525) + +``` +FSPL = 20 * log10(d_km) + 20 * log10(f_GHz) + 92.45 (dB) +``` + +### Fresnel Zone Radius + +``` +r_fresnel = sqrt(lambda * d1 * d2 / (d1 + d2)) + +lambda = 0.3 / f_GHz (meters) +``` + +### Earth Bulge + +``` +bulge = (d1 * d2) / (2 * K * 6371000) + +K: effective earth radius factor (standard = 4/3) +``` + +### Effective K-Factor from Surface N + +``` +dN_est = -40 - (N_surface - 315) * 0.25 +K = 1 / (1 + 6371 * dN_est * 1e-6) +K clamped to [0.5, 5.0] +``` --- -## The Single Most Important Thing: Moisture Affects Each Band Differently +## Part 2: Key Empirical Findings -This is the biggest insight from analyzing 58,000 contest QSOs. **Humidity helps 10 GHz but hurts everything above it.** If you only remember one thing from this document, remember this. +These findings drive the scoring model's design. Each contradicts or refines assumptions from simpler models. -### Why? +### Finding 1: Humidity Effect Reverses by Frequency -There are two competing effects when moisture is in the air: +The most important discovery. At 10 GHz, more moisture = longer paths (refractivity dominates, absorption negligible). At 24+ GHz, more moisture = shorter paths (absorption dominates). -1. **Refractivity** — Water vapor bends radio waves. More moisture means more bending, which helps signals follow the curve of the earth. This is how ducting works. This effect matters at every frequency. +**10 GHz — humidity helps (N=52,456 QSOs):** +| Abs. Humidity | Avg Dist | P90 Dist | +|---------------|----------|----------| +| 5-8 g/m^3 | 193 km | 342 km | +| 11-14 g/m^3 | 215 km | 383 km | +| 17+ g/m^3 | 230 km | 519 km | -2. **Absorption** — Water molecules absorb microwave energy. There's a strong absorption peak at 22.235 GHz (a natural resonance frequency of the water molecule). Frequencies near this peak lose a lot of energy to moisture. This effect gets worse the closer you are to 22 GHz. +**24 GHz — humidity hurts (N=3,439 QSOs):** +| Abs. Humidity | Avg Dist | P90 Dist | +|---------------|----------|----------| +| 5-8 g/m^3 | 115 km | 154 km | +| 11-14 g/m^3 | 105 km | 174 km | +| 17+ g/m^3 | 53 km | 103 km | -At **10 GHz**, absorption from water vapor is tiny — about 0.01 dB per kilometer. The refractivity benefit from extra moisture far outweighs this small loss. So humid air actually produces longer-distance contacts at 10 GHz. Our data shows average distances increasing from 193 km in dry air to 230 km in very humid air. +**47 GHz — humidity hurts, less severely (N=576 QSOs):** +| Abs. Humidity | Avg Dist | P90 Dist | +|---------------|----------|----------| +| 0-5 g/m^3 | 191 km | 234 km | +| 11-14 g/m^3 | 71 km | 114 km | -At **24 GHz**, you're sitting right next to that 22.235 GHz water vapor absorption line. Humidity now costs you roughly 0.10 dB per kilometer — ten times more than at 10 GHz. The absorption penalty crushes any refractivity benefit. Dry air is essential. Average distances drop from 115 km in dry air to just 53 km in very humid conditions. +### Finding 2: Wind Penalty Is Overrated -At **47 GHz** and above, humidity still hurts (you're on the far side of the water vapor line, and there's another one at 183 GHz), but the oxygen absorption around 60 GHz starts to matter too — and there's nothing you can do about oxygen. It's always there, always absorbing, regardless of weather. +Data shows no meaningful penalty for wind on achieved distance: + +| Wind | 10G Avg | 24G Avg | 47G Avg | +|------|---------|---------|---------| +| Calm (0-3 kts) | 216 | 84 | -- | +| Light (3-7 kts) | 214 | 100 | 77 | +| Moderate (7-12 kts) | 220 | 113 | 77 | + +Wind may reduce inversion quality but also creates boundary-layer dynamics that can enhance propagation. Weight reduced from 18% to 8%. + +### Finding 3: Low Pressure Correlates with Longer Distances + +Contradicts the original model's high-pressure = good assumption: + +| Pressure | 10G Avg | 10G P90 | 24G Avg | 47G Avg | +|----------|---------|---------|---------|---------| +| <1010 | 262 | 506 | 119 | 111 | +| 1015-1020 | 217 | 383 | 97 | 74 | +| 1025+ | 196 | 354 | 122 | -- | + +Low pressure systems bring frontal boundaries with strong temperature/moisture gradients that create inversions and ducts. The key is gradient structure, not absolute pressure. + +### Finding 4: Boundary Layer Depth Sweet Spot + +Moderate BL depth (500-1000m) produces the best results across all bands — indicating an elevated inversion high enough to trap signals but not so deep that full convective mixing has occurred. + +### Finding 5: Binary Duct Detection Is Weak + +Simple "duct detected yes/no" from soundings shows negligible correlation with distance (avg dist: 214 vs 216 km at 10 GHz). This is because soundings are 12-hourly point samples while conditions evolve continuously, and duct characteristics (strength, height) matter more than binary presence. + +### Finding 6: Diurnal Signal Variation Sets a Noise Floor + +Commercial link data shows 1-5 dB daily variation even on perfectly clear, stable days. The algorithm should convey that even an "EXCELLENT" score has +/-2-3 dB inherent uncertainty. + +### Finding 7: LOS vs Beyond-LOS Regimes Are Inverted + +On short LOS paths (3-7 km), sub-refractive conditions (dN/dh > -40/km) produce the best signal — minimal multipath, clean beam coupling. On long beyond-LOS paths (50-500+ km), enhanced refraction/ducting is essential. The algorithm must handle both regimes. --- -## The Nine Scoring Factors +## Part 3: Band Configuration -The algorithm combines nine measurements into a single score. Each factor is scored 0-100, then combined using weights that reflect how much each one actually matters (based on what the real QSO data tells us, not theory alone). - -### 1. Humidity (22% of the score) - -Absolute humidity — the actual grams of water per cubic meter of air — is what matters. Not relative humidity, not dewpoint alone. The algorithm computes absolute humidity from temperature and dewpoint using standard physics. - -**For 10 GHz:** Moderate to high humidity (10-18 g/m³) scores well because it increases the atmosphere's ability to bend signals. Very dry air (below 4 g/m³) scores poorly — not enough refractivity to create ducting. Extremely humid air (above 22 g/m³) scores slightly lower because of scintillation (signal shimmer from turbulent moisture). - -**For 24 GHz and up:** Dry air scores best. The drier, the better. Each band has a sensitivity multiplier — 24 GHz is the worst because it's closest to the water vapor absorption line. 241 GHz is extremely sensitive too because it sits between two more water vapor lines (183 and 325 GHz). - -### 2. Time of Day (18% of the score) - -Temperature inversions are the engine of microwave propagation. During the day, the sun heats the ground, which heats the air, which rises — destroying any stable layers. At night, the ground cools by radiation, creating a cold layer near the surface with warmer air above it. This inversion traps radio waves. - -The cycle goes like this: - -- **Late evening (after sunset):** The ground starts cooling. An inversion begins forming. -- **Overnight:** The inversion strengthens as the ground keeps radiating heat. -- **Pre-dawn:** The inversion is strong. Conditions are good and getting better. -- **Sunrise to about 90 minutes after:** Peak inversion strength. This is the golden window. -- **Mid-morning:** The rising sun starts warming the ground. Convection begins eroding the inversion from below. -- **Afternoon:** The inversion is gone. Full convective mixing. Worst conditions of the day. - -The algorithm uses a per-month sunrise table (for Central time) and scores each moment relative to sunrise. The best score (100) goes to the window from about 30 minutes before sunrise through 90 minutes after. The worst score (18) goes to the afternoon hours after the inversion is fully destroyed. - -### 3. Temperature-Dewpoint Depression (14% of the score) - -The spread between temperature and dewpoint tells you how dry the air is aloft. A large spread (say, temperature 80°F and dewpoint 50°F — a 30-degree depression) means dry air overhead, which favors stable layering. - -**For 10 GHz:** A moderate depression (8-14°F) actually scores best — you want enough moisture for refractivity but not so much that you're in fog. Very large depressions (very dry air) score lower because refractivity drops. - -**For 24 GHz and up:** Bigger depression = drier air = less absorption = higher score. Simple. - -### 4. Season (10% of the score) - -Seasonal patterns are really just a proxy for long-term humidity trends. In the central US: - -- **November through February:** Best months. Cold, dry air. Low absolute humidity. Strong radiation inversions on clear nights. -- **July and August:** Worst months. Gulf moisture makes the boundary layer a soup of water vapor. Even the inversions that form are moisture-laden. -- **Spring and Fall:** Transitional. September is better than June because summer moisture starts retreating. - -Every band follows this basic pattern, but the higher frequencies have even steeper seasonal penalties in summer because they're more sensitive to moisture. - -### 5. Sky Condition (10% of the score) - -Cloud cover matters mostly because clouds indicate moisture and vertical mixing: - -- **Clear/Few clouds:** Score 88-100. Clear skies allow maximum radiative cooling at night, building stronger inversions. -- **Scattered:** Score 60. Some convection, some stability. -- **Broken/Overcast:** Score 5-25. Clouds trap heat, preventing the ground from cooling. Inversions can't form as easily. Often indicates an active weather pattern with vertical mixing. -- **Vertical Visibility (fog):** Score 5. Very high near-surface moisture. Bad for everything above 10 GHz. - -The data shows sky condition matters less at 10 GHz (213 km average distance regardless of clouds) and more at 24/47 GHz where the associated moisture matters. - -### 6. Rain (8% of the score) - -Rain kills microwave signals. The higher the frequency, the worse it gets. At 10 GHz, moderate rain (10 mm/hr) costs you about 0.2 dB per kilometer — annoying but survivable. At 47 GHz, the same rain costs 1.6 dB/km. At 75 GHz and above, even light rain (4 mm/hr) adds over 1 dB/km, which effectively destroys any path beyond a few kilometers. - -The algorithm computes rain attenuation using the ITU rain model (which uses two constants, k and alpha, that vary by frequency) and scores it as a simple penalty — no rain = perfect score, increasing rain = decreasing score until the path is destroyed. - -### 7. Wind (8% of the score) - -Wind was originally weighted at 18% in the old algorithm. The data doesn't support that. Across 50,000+ contacts at 10 GHz, calm winds averaged 216 km and moderate winds averaged 220 km — essentially no difference. - -Wind does matter for two things: -- Very strong winds (above 20 knots) create turbulent scintillation that degrades signals -- Calm air allows inversions to form undisturbed - -But the effect is much smaller than originally assumed. The algorithm now gives a mild penalty for strong winds and otherwise stays out of the way. - -### 8. Pressure (5% of the score) - -Barometric pressure, by itself, is a weak predictor. The original algorithm scored rising pressure as good and falling as bad, based on the theory that high pressure = stable atmosphere. - -The data tells a more nuanced story. Low pressure (below 1010 hPa) actually correlates with the longest 10 GHz distances — 262 km average versus 196 km at high pressure (above 1025 hPa). Why? Because frontal boundaries create dramatic temperature and moisture gradients that produce strong inversions and ducts. - -The revised algorithm cares more about pressure *change* (the gradient) than the absolute value: -- Rapidly rising: Post-frontal clearing — inversions forming, score 80 -- Slowly falling: Approaching front — possible ducting, score 65 -- Rapidly falling: Active weather — unstable, score 45 - -### 9. Refractivity / Sounding Data (5% of the score) - -When upper-air sounding data is available (from weather balloon launches at 00Z and 12Z), the algorithm gets its best look at the actual vertical structure of the atmosphere. - -The key measurement is the **modified refractivity gradient** (how the atmosphere's bending power changes with height). In a standard atmosphere, this gradient is about -40 M-units per kilometer. When the gradient drops below -157 M-units per kilometer, a **duct** has formed — a layer that traps radio waves and guides them along the earth's surface. - -The algorithm also looks at **boundary layer depth** — the height of the lowest mixing layer. A depth of 500-1000 meters is the sweet spot across all bands. This corresponds to a morning condition where the nocturnal inversion has been lifted by early heating but hasn't been destroyed by full convective mixing. - -Since soundings only happen twice a day, this factor gets a modest weight. When sounding data isn't available, it defaults to a neutral score of 50. +```elixir +@band_configs %{ + 10_000 => %{ + label: "10 GHz", + o2_db_km: 0.008, + h2o_coeff: 0.0005, + humidity_effect: :beneficial, # More moisture = more refractivity = longer paths + humidity_penalty: 0.0, + rain_k: 0.010, rain_alpha: 1.28, + seasonal_base: %{1 => 88, 2 => 84, 3 => 72, 4 => 62, 5 => 55, + 6 => 42, 7 => 28, 8 => 28, 9 => 52, 10 => 68, + 11 => 96, 12 => 88}, + seasonal_adj: %{}, + typical_range_km: 200, + extended_range_km: 500, + exceptional_range_km: 1000 + }, + 24_000 => %{ + label: "24 GHz", + o2_db_km: 0.015, + h2o_coeff: 0.012, # Validated by commercial link data + humidity_effect: :harmful, # 22.235 GHz H2O line shoulder + humidity_penalty: 1.6, + rain_k: 0.070, rain_alpha: 1.07, + seasonal_base: %{1 => 88, 2 => 84, 3 => 72, 4 => 62, 5 => 51, + 6 => 34, 7 => 18, 8 => 18, 9 => 48, 10 => 68, + 11 => 96, 12 => 88}, + seasonal_adj: %{5 => -4, 6 => -8, 7 => -10, 8 => -10, 9 => -4}, + typical_range_km: 100, + extended_range_km: 250, + exceptional_range_km: 500 + }, + 47_000 => %{ + label: "47 GHz", + o2_db_km: 0.045, + h2o_coeff: 0.003, + humidity_effect: :harmful, + humidity_penalty: 1.0, # Window band, moderate H2O sensitivity + rain_k: 0.187, rain_alpha: 0.93, + seasonal_base: %{1 => 90, 2 => 88, 3 => 78, 4 => 68, 5 => 55, + 6 => 38, 7 => 22, 8 => 22, 9 => 48, 10 => 74, + 11 => 96, 12 => 90}, + seasonal_adj: %{}, + typical_range_km: 70, + extended_range_km: 150, + exceptional_range_km: 300 + }, + 68_000 => %{ + label: "68 GHz", + o2_db_km: 0.90, # 60 GHz O2 band wing — validated by link data + h2o_coeff: 0.007, # Measured: ~0.1 dB/km per g/m^3 on 2.8 km path + humidity_effect: :harmful, + humidity_penalty: 1.4, + rain_k: 0.310, rain_alpha: 0.86, + seasonal_base: %{1 => 90, 2 => 88, 3 => 78, 4 => 65, 5 => 50, + 6 => 32, 7 => 18, 8 => 18, 9 => 44, 10 => 70, + 11 => 92, 12 => 90}, + seasonal_adj: %{}, + typical_range_km: 40, + extended_range_km: 80, + exceptional_range_km: 150 + }, + 75_000 => %{ + label: "75 GHz", + o2_db_km: 0.012, + h2o_coeff: 0.006, + humidity_effect: :harmful, + humidity_penalty: 1.2, + rain_k: 0.345, rain_alpha: 0.84, + seasonal_base: %{1 => 90, 2 => 90, 3 => 80, 4 => 68, 5 => 55, + 6 => 38, 7 => 22, 8 => 22, 9 => 48, 10 => 74, + 11 => 96, 12 => 90}, + seasonal_adj: %{}, + typical_range_km: 50, + extended_range_km: 120, + exceptional_range_km: 250 + }, + 122_000 => %{ + label: "122 GHz", + o2_db_km: 0.80, # 118.75 GHz O2 wing — weather independent + h2o_coeff: 0.010, + humidity_effect: :harmful, + humidity_penalty: 1.0, + rain_k: 0.498, rain_alpha: 0.77, + seasonal_base: %{1 => 92, 2 => 90, 3 => 78, 4 => 62, 5 => 45, + 6 => 28, 7 => 15, 8 => 15, 9 => 38, 10 => 68, + 11 => 92, 12 => 92}, + seasonal_adj: %{}, + typical_range_km: 30, + extended_range_km: 80, + exceptional_range_km: 140 + }, + 134_000 => %{ + label: "134 GHz", + o2_db_km: 0.08, + h2o_coeff: 0.015, + humidity_effect: :harmful, + humidity_penalty: 1.3, + rain_k: 0.520, rain_alpha: 0.75, + seasonal_base: %{1 => 92, 2 => 90, 3 => 78, 4 => 65, 5 => 48, + 6 => 30, 7 => 18, 8 => 18, 9 => 42, 10 => 70, + 11 => 92, 12 => 92}, + seasonal_adj: %{}, + typical_range_km: 40, + extended_range_km: 100, + exceptional_range_km: 160 + }, + 241_000 => %{ + label: "241 GHz", + o2_db_km: 0.08, + h2o_coeff: 0.30, # Extreme H2O sensitivity (183/325 GHz lines) + humidity_effect: :harmful, + humidity_penalty: 3.0, + rain_k: 0.550, rain_alpha: 0.70, + seasonal_base: %{1 => 95, 2 => 92, 3 => 75, 4 => 55, 5 => 35, + 6 => 15, 7 => 8, 8 => 8, 9 => 30, 10 => 65, + 11 => 95, 12 => 95}, + seasonal_adj: %{}, + typical_range_km: 10, + extended_range_km: 50, + exceptional_range_km: 115 + } +} +``` --- -## Putting It All Together +## Part 4: Scoring Functions (Beyond-LOS Regime) -The nine factor scores get multiplied by their weights and added up: +All scores return 0-100. The beyond-LOS regime is the primary use case for ham radio propagation prediction. -| Factor | Weight | -|--------|--------| -| Humidity | 22% | -| Time of Day | 18% | -| T-Td Depression | 14% | -| Season | 10% | -| Sky Condition | 10% | -| Rain | 8% | -| Wind | 8% | -| Pressure | 5% | -| Refractivity | 5% | +### 1. Humidity Score — Frequency-Dependent -A perfect day — dry (or humid at 10 GHz), sunrise, clear skies, calm winds, winter, no rain, strong inversion — would score close to 100. A summer afternoon with thunderstorms and 20 g/m³ of moisture would score close to 0. +The critical insight: moisture helps at 10 GHz (refractivity) and hurts at 24+ GHz (absorption). + +```elixir +def score_humidity(abs_humidity_gm3, band_config) do + case band_config.humidity_effect do + :beneficial -> + # 10 GHz: more moisture = higher surface N = more beam bending + # Extreme humidity risks scintillation + cond do + abs_humidity_gm3 < 4 -> 55 # Very dry — poor refractivity + abs_humidity_gm3 < 7 -> 70 # Dry + abs_humidity_gm3 < 10 -> 82 # Moderate + abs_humidity_gm3 < 14 -> 90 # Good refractivity + abs_humidity_gm3 < 18 -> 95 # Excellent refractivity + abs_humidity_gm3 < 22 -> 88 # High — scintillation onset + true -> 75 # Tropical — scintillation risk + end + + :harmful -> + # 24+ GHz: H2O absorption dominates + # Penalty factor scales by band (1.0 for 47G window, 1.6 for 24G near line, 3.0 for 241G) + r = abs_humidity_gm3 * band_config.humidity_penalty + cond do + r <= 6 -> 100 + r <= 9 -> round(95 - (r - 6) / 3 * 20) + r <= 13 -> round(75 - (r - 9) / 4 * 30) + r <= 18 -> round(45 - (r - 13) / 5 * 35) + true -> max(0, round(10 - (r - 18) * 2)) + end + end +end +``` + +### 2. Time of Day Score — Inversion Lifecycle + +The strongest diurnal predictor in the data. Dawn shows the highest P90 distances; late evening shows elevated averages. + +```elixir +@sunrise_table [7.4, 7.3, 7.0, 6.7, 6.35, 6.25, + 6.35, 6.65, 6.9, 7.1, 7.35, 7.45] + +def score_time_of_day(utc_hour, utc_minute, month) do + offset = if month >= 3 and month <= 10, do: -5, else: -6 # CDT/CST + local = rem(utc_hour + utc_minute / 60 + offset + 24, 24) + sunrise = Enum.at(@sunrise_table, month - 1) + d = local - sunrise # hours relative to sunrise + + cond do + d >= -1.5 and d <= 1.5 -> + {100, "Peak — inversion maximum"} + + d > 1.5 and d <= 3.0 -> + {78, "Good — inversion eroding"} + + d > -3.0 and d < -1.5 -> + {82, "Pre-dawn — inversion building"} + + d > 3.0 and d <= 6.0 -> + {38, "Marginal — boundary layer mixing"} + + local >= 20.0 or local <= 1.0 -> + {72, "Evening — cooling, inversion reforming"} + + d > 6.0 -> + {18, "Afternoon — full convective mixing"} + + true -> + {55, "Night — gradual cooling"} + end +end +``` + +### 3. Temperature-Dewpoint Depression — Frequency-Split + +Large depression = dry aloft = favorable for 24+ GHz. Small depression = moist = favorable for 10 GHz refractivity (but near-saturation risks fog). + +```elixir +def score_td_depression(temp_f, dewpoint_f, band_config) do + dep = temp_f - dewpoint_f + + case band_config.humidity_effect do + :beneficial -> + cond do + dep < 3 -> 40 # Near saturation — fog/scattering risk + dep < 8 -> 75 # Moist — good refractivity + dep < 14 -> 85 # Moderate — balanced + dep < 22 -> 70 # Dry — reduced refractivity + true -> 55 # Very dry — poor refractivity + end + + :harmful -> + cond do + dep > 22 -> 96 # Very dry aloft + dep > 14 -> 80 # Good stability + dep > 8 -> 60 # Moderate + dep > 4 -> 38 # Marginal + true -> 18 # Humid aloft + end + end +end +``` + +### 4. Sky Cover Score + +Data shows modest impact at 24/47 GHz, near-zero at 10 GHz. VV (vertical visibility / fog) is a moderate penalty due to near-surface moisture content. + +```elixir +def score_sky(condition) do + case condition do + c when c in ["CLR", "SKC"] -> 100 + "FEW" -> 88 + "SCT" -> 60 + "BKN" -> 25 + "OVC" -> 5 + "VV" -> 5 + _ -> 50 + end +end +``` + +### 5. Season Score + +Per-band lookup with optional adjustments. 24 GHz gets additional summer penalties due to Gulf moisture. + +```elixir +def score_season(month, band_config) do + base = Map.get(band_config.seasonal_base, month, 50) + adj = Map.get(band_config.seasonal_adj, month, 0) + max(0, min(100, base + adj)) +end +``` + +### 6. Wind Score — Reduced Weight + +Data shows minimal impact on achieved distance. Retain mild penalty only for very high winds (turbulent scintillation). + +```elixir +def score_wind(speed_kts) do + cond do + speed_kts < 5 -> 100 + speed_kts < 10 -> 90 + speed_kts < 15 -> 75 + speed_kts < 20 -> 55 + speed_kts < 25 -> 35 + true -> 15 + end +end +``` + +### 7. Rain Score + +Not validated by measured data. Based on ITU-R P.838-3 attenuation per km. At 10 GHz, moderate rain is tolerable. Above 75 GHz, even light rain effectively kills the path. + +```elixir +def score_rain(rain_rate_mmhr, band_config) do + if rain_rate_mmhr == nil or rain_rate_mmhr == 0 do + 100 + else + gamma = band_config.rain_k * :math.pow(rain_rate_mmhr, band_config.rain_alpha) + cond do + gamma < 0.1 -> 95 + gamma < 0.5 -> 75 + gamma < 1.0 -> 50 + gamma < 2.0 -> 25 + gamma < 5.0 -> 10 + true -> 0 + end + end +end +``` + +### 8. Pressure Score — Gradient-Focused + +Data contradicts the original high-pressure-is-good model. Frontal boundaries (changing pressure) create the strongest refractive gradients. Rising post-frontal pressure and slowly falling pre-frontal pressure both score well. + +```elixir +def score_pressure(current_mb, previous_mb) do + case previous_mb do + nil -> + # No trend — mild scoring on absolute value + cond do + current_mb > 1025 -> 55 # Strong ridge + current_mb > 1018 -> 65 # Mild high + current_mb > 1010 -> 60 # Normal + current_mb > 1005 -> 55 # Low + true -> 40 # Very low — active weather + end + + prev -> + delta = current_mb - prev + cond do + delta > 2.5 -> 80 # Rising rapidly — post-frontal clearing + delta > 0.8 -> 70 # Rising — stabilizing + delta > -0.5 -> 60 # Steady — neutral + delta > -2.0 -> 65 # Falling slowly — approaching front, duct possible + true -> 45 # Falling rapidly — active weather + end + end +end +``` + +### 9. Refractivity Score — When Sounding/HRRR Data Available + +Best predictor when available, but only available from 12-hourly soundings or HRRR model. Binary duct detection is weak; refractivity gradient and BL depth are stronger signals. + +```elixir +def score_refractivity(sounding_or_hrrr, band_config) do + cond do + sounding_or_hrrr == nil -> 50 # No data — neutral + + sounding_or_hrrr.min_dn_dh < -500 -> + # Super-refraction — ducting probable + case band_config.humidity_effect do + :beneficial -> 98 # 10 GHz benefits most from ducting + :harmful -> 85 # Higher bands benefit but absorption limits range + end + + sounding_or_hrrr.min_dn_dh < -200 -> + 80 # Enhanced refraction + + sounding_or_hrrr.bl_depth_m != nil and + sounding_or_hrrr.bl_depth_m >= 500 and + sounding_or_hrrr.bl_depth_m <= 1000 -> + 78 # Sweet-spot BL depth — elevated inversion + + sounding_or_hrrr.min_dn_dh < -100 -> + 65 # Mild enhancement + + true -> + 50 # Standard conditions + end +end +``` --- -## What the Score Means for Each Band +## Part 5: Composite Score -The same score translates to very different achievable distances depending on frequency. Higher frequencies face more path loss and more absorption, so the same "excellent" conditions produce shorter paths: +### Weights -| Score | 10 GHz | 24 GHz | 47 GHz | 75 GHz | -|-------|--------|--------|--------|--------| -| Excellent (80+) | 400-1000+ km | 200-500 km | 120-300 km | 80-200+ km | -| Good (65-79) | 250-400 km | 120-200 km | 80-120 km | 50-80 km | -| Marginal (50-64) | 150-250 km | 70-120 km | 50-80 km | 30-50 km | -| Poor (33-49) | 80-150 km | 40-70 km | 25-50 km | 15-30 km | -| Negligible (0-32) | <80 km | <40 km | <25 km | <15 km | +| Factor | Weight | Rationale | +|--------|--------|-----------| +| Humidity | 22% | Dominant variable, but split role by frequency | +| Time of Day | 18% | Strongest diurnal predictor in QSO data | +| Td Depression | 14% | Proxy for humidity aloft — strong signal | +| Sky Cover | 10% | Modest effect, mainly at higher frequencies | +| Season | 10% | Long-term baseline | +| Wind | 8% | Data shows minimal impact; penalty only for extremes | +| Rain | 8% | Critical for 24+ GHz paths | +| Pressure | 5% | Weak standalone predictor | +| Refractivity | 5% | Best predictor when available, but often unavailable | -At 122 GHz and above, the oxygen absorption line at 118.75 GHz adds a fixed ~0.8 dB/km penalty that no weather can fix. These bands are realistically limited to about 30-50 km for typical contacts and 80-140 km under exceptional conditions. +```elixir +def composite_score(factors) do + round( + factors.humidity * 0.22 + + factors.time_of_day * 0.18 + + factors.td_depression * 0.14 + + factors.sky * 0.10 + + factors.season * 0.10 + + factors.wind * 0.08 + + factors.rain * 0.08 + + factors.pressure * 0.05 + + factors.refractivity * 0.05 + ) +end +``` -At 241 GHz, you're between two water vapor lines (183 and 325 GHz). Even in extremely dry winter air, you're losing about 2 dB per kilometer. The practical limit is about 10-50 km for most work, with the 114 km record requiring near-perfect winter conditions and possibly elevated terrain. +### Score Tiers with Per-Band Range Estimates + +| Score | Label | 10G | 24G | 47G | 75G | +|-------|-------|-----|-----|-----|-----| +| 80-100 | EXCELLENT | 400-1000+ km | 200-500 km | 120-300 km | 80-200+ km | +| 65-79 | GOOD | 250-400 km | 120-200 km | 80-120 km | 50-80 km | +| 50-64 | MARGINAL | 150-250 km | 70-120 km | 50-80 km | 30-50 km | +| 33-49 | POOR | 80-150 km | 40-70 km | 25-50 km | 15-30 km | +| 0-32 | NEGLIGIBLE | <80 km | <40 km | <25 km | <15 km | + +| Color | Hex | +|-------|-----| +| EXCELLENT | #00ffa3 | +| GOOD | #7dffd4 | +| MARGINAL | #ffe566 | +| POOR | #ff9044 | +| NEGLIGIBLE | #ff4f4f | --- -## The 6-Hour Forecast +## Part 6: LOS Regime Scoring -The algorithm can't predict the future, but it can project current trends forward using two things: +For known fixed links or short paths with confirmed Fresnel clearance. Key difference: sub-refraction is neutral/beneficial (minimal multipath), and gaseous absorption is the primary variable. -### 1. The Diurnal Cycle +### LOS Refractivity Score -The daily temperature/humidity cycle is highly predictable. If it's 3 AM and conditions are good, the algorithm knows that sunrise is coming and the inversion will peak in a couple of hours — so the score will improve. If it's noon, the algorithm knows the afternoon convective mixing will keep things poor for several more hours. +```elixir +def score_refractivity_los(dn_dh) do + cond do + dn_dh > 0 -> 60 # Strong sub-refraction — unusual but not harmful + dn_dh > -30 -> 85 # Moderate sub-refraction — stable, clean signal + dn_dh > -40 -> 75 # Near standard + dn_dh > -80 -> 60 # Enhanced — multipath onset + dn_dh > -157 -> 45 # Strong enhancement — multipath likely + true -> 30 # Super-refraction — significant multipath fading + end +end +``` -The forecast blends the current trend (what temperature and dewpoint have been doing over the last 3 hours) with the expected diurnal pattern: -- Short-term (1-2 hours): Current trend dominates -- Medium-term (3-4 hours): Blend of trend and diurnal model -- Longer-term (5-6 hours): Diurnal model dominates +### LOS Surface N Score -### 2. Forecast Weather Data +Higher N often means more moisture = more absorption at 24+ GHz. Validated by link data: N < 310 gave best 68 GHz signal, N > 340 gave worst. -When NWS forecast data is available (cloud cover, precipitation probability, wind forecasts), the algorithm uses it for the factors that can't be projected from surface trends alone — mainly rain and sky condition. +```elixir +def score_surface_n(n_value, band_config) do + case band_config.humidity_effect do + :beneficial -> + cond do + n_value > 350 -> 90 + n_value > 330 -> 80 + n_value > 315 -> 65 + n_value > 300 -> 50 + true -> 35 + end -### Confidence + :harmful -> + cond do + n_value < 300 -> 90 + n_value < 315 -> 80 + n_value < 330 -> 65 + n_value < 345 -> 50 + true -> 35 + end + end +end +``` -Predictions get less reliable with time. The algorithm reports confidence alongside each forecast point: +### LOS vs Beyond-LOS Selection -| Horizon | Confidence | What It Means | -|---------|------------|---------------| -| +1 hour | 85% | Pretty reliable. Atmospheric conditions don't change fast. | -| +2 hours | 75% | Still good. The diurnal cycle is predictable. | -| +3 hours | 60% | Okay. Unexpected fronts or convection could change things. | -| +4 hours | 50% | Coin flip territory for details, but general trend is still useful. | -| +5-6 hours | 30-40% | General direction only. A lot can change. | +```elixir +def compute_score(conditions, band_config, path_type \\ :beyond_los) do + base_factors = %{ + humidity: score_humidity(conditions.abs_humidity, band_config), + wind: score_wind(conditions.wind_speed_kts), + sky: score_sky(conditions.sky_condition), + time_of_day: score_time_of_day(conditions.utc_hour, conditions.utc_minute, conditions.month) |> elem(0), + td_depression: score_td_depression(conditions.temp_f, conditions.dewpoint_f, band_config), + season: score_season(conditions.month, band_config), + pressure: score_pressure(conditions.slp, conditions.prev_slp), + rain: score_rain(conditions.rain_rate, band_config) + } -Even the best prediction has about ±2-3 dB of inherent noise — the atmosphere is never perfectly still, equipment has thermal variation, and multipath geometry constantly shifts. The algorithm communicates this uncertainty rather than pretending to have precision it doesn't have. + factors = case path_type do + :beyond_los -> + Map.put(base_factors, :refractivity, + score_refractivity(conditions.sounding, band_config)) + + :los -> + Map.put(base_factors, :refractivity, + score_refractivity_los(conditions.dn_dh)) + end + + %{score: composite_score(factors), factors: factors} +end +``` --- -## Band-by-Band Guide +## Part 7: Link Budget -### 10 GHz (3 cm) +For point-to-point path analysis with known station parameters. -**The workhorse band.** 52,000+ QSOs in our dataset, with contacts out to 1,609 km. +### EIRP -**What makes it special:** At 10 GHz, the atmosphere is nearly transparent. Gaseous absorption is negligible. What matters is whether the atmosphere bends the signal — and moisture helps with that. This is the one band where humid conditions are actually *good* for long-distance work. +``` +eirp_dbm = tx_power_dbm + tx_antenna_dbi - feed_loss_db +``` -**Best conditions:** Moderate to high humidity, temperature inversions (especially radiation inversions overnight through early morning), stable atmosphere with a boundary layer depth of 500-1000 meters. Marine ducting along coastlines can produce 1,000+ km paths. +### Receiver Sensitivity -**Worst conditions:** Very dry air (low refractivity), afternoon convective mixing, no inversion structure. +``` +sensitivity_dbm = -174 + noise_figure_db + 10 * log10(bandwidth_hz) -**Rain:** Not a significant concern. Even moderate rain only costs about 0.2 dB/km. +CW: bandwidth = 500 Hz +SSB: bandwidth = 2700 Hz +``` -**Key insight:** 10 GHz propagation is about atmosphere structure, not absorption. Focus on inversions and refractivity, not humidity levels. +### Total Path Loss -### 24 GHz (1.2 cm) +``` +total_loss = FSPL + gaseous_absorption + rain_attenuation + diffraction_loss - duct_enhancement -**The humidity-sensitive band.** 3,400+ QSOs, distances to 710 km (the longest was rain scatter, a completely different mechanism). +gaseous_absorption = (o2_db_km + h2o_coeff * rho) * distance_km +rain_attenuation = gamma_R * distance_km * rain_effective_fraction +``` -**What makes it special:** 24 GHz sits right next to the 22.235 GHz water vapor absorption line. No other ham band is this close to a major atmospheric absorption feature. Every gram of water vapor in the air costs you about ten times more signal than at 10 GHz. +### Duct Enhancement (Beyond-LOS Only) -**Best conditions:** Very dry air (below 8 g/m³ absolute humidity), cold season (November through March), clear skies, strong pre-dawn inversions. +Calibrated against confirmed contacts: -**Worst conditions:** Summer. The Gulf moisture floods the boundary layer, and there's simply too much water vapor to overcome. Summer range is roughly half of winter potential. +```elixir +def duct_enhancement_db(prop_score) do + cond do + prop_score >= 80 -> -14 # 14 dB improvement + prop_score >= 65 -> -10 + prop_score >= 50 -> -6 + prop_score >= 33 -> -2 + true -> 0 + end +end +``` -**Rain:** A serious concern. 10 mm/hr rain adds nearly 1 dB/km. Moderate rain can cut achievable range in half. +### Knife-Edge Diffraction (ITU-R P.526) -**Rain scatter:** At 24 GHz, heavy rain cells can scatter signals to very long distances. One 710 km QSO was explicitly labeled as rain scatter. This is a completely different propagation mechanism from ducting — you're bouncing signals off the rain cell rather than through it. +```elixir +def knife_edge_loss(v) do + cond do + v <= -0.7787 -> 0 # Clear + v <= 0 -> -20 * :math.log10(0.5 - 0.62 * v) + v <= 1 -> -20 * :math.log10(0.5 * :math.exp(-0.95 * v)) + v <= 2.4 -> + inner = max(0, 0.1184 - (0.38 - 0.1 * v) ** 2) + -20 * :math.log10(0.4 - :math.sqrt(inner)) + true -> 20 * :math.log10(v) + 13.0 # Asymptotic + end +end +``` -**Key insight:** Track absolute humidity obsessively. A drying trend is the most actionable prediction for this band. +### Success Probability -### 47 GHz (6 mm) +```elixir +def margin_to_success(margin_db, prop_score) do + margin_pct = cond do + margin_db <= 0 -> 0 + margin_db <= 10 -> margin_db / 10 * 20 + margin_db <= 15 -> 20 + (margin_db - 10) / 5 * 20 + margin_db <= 20 -> 40 + (margin_db - 15) / 5 * 20 + margin_db <= 25 -> 60 + (margin_db - 20) / 5 * 20 + margin_db <= 30 -> 80 + (margin_db - 25) / 5 * 20 + true -> 100 + end -**The atmospheric window.** 576 QSOs, distances to 343 km. + # Propagation modulation: score 100 -> x1.30, score 50 -> x1.00, score 0 -> x0.70 + prop_factor = 0.70 + (prop_score / 100) * 0.60 + max(0, min(99, round(margin_pct * prop_factor))) +end +``` -**What makes it special:** 47 GHz sits in a gap between the 22 GHz water vapor line and the 60 GHz oxygen band. It's less humidity-sensitive than 24 GHz, but oxygen absorption starts to become noticeable (~0.045 dB/km, weather-independent). +### Note: Antenna Height & Duct Coupling Geometry -**Best conditions:** Very dry air, clear skies, strong inversions. Very dry winter conditions can produce surprising distances (246 km documented). +Antenna height and dish elevation angle affect how efficiently a station couples into an atmospheric duct. This is a real physical effect but is **second-order** to duct characteristics at the ranges this model targets (50-1000+ km). -**Worst conditions:** Any precipitation, high humidity, summer months. +**Why it's not in the scoring model:** +- At >300 km, the duct's own refractive gradient (k-factor) dominates over all antenna geometry. The required aim angle to graze a duct converges toward 0° regardless of antenna height. +- Antenna height differences in the 15-21m range (typical amateur stations) shift beam geometry by ~0.001° at long range — well within the ±2-3 dB noise floor from diurnal variation. +- The primary benefit of antenna height (50+ ft) is clearing local obstructions and ground clutter in the near field (0-20 km), not geometric coupling to the duct layer. +- VE4MA (50 ft, flat prairie) and W5LUA (70 ft, suburban) achieve similar range classes, confirming duct geometry is the dominant term. -**Key insight:** Ducting events are the ONLY way to exceed about 150 km. Without ducting, free-space loss plus gas absorption limits things. Monitor for inversions during the early morning window. - -### 68 GHz (V-band edge) - -**The oxygen absorption fringe.** Not a common ham band, but our terrestrial link data at 68 GHz provides valuable insight. - -**What makes it special:** You're on the edge of the massive 60 GHz oxygen absorption complex. O₂ absorption is about 0.5-1.0 dB/km — weather-independent, always present. On top of that, water vapor adds its own penalty. - -**Best conditions:** Cold, dry air reduces the water vapor component. But nothing fixes the oxygen problem. - -**Practical limits:** About 40 km for typical contacts, maybe 80-150 km under exceptional conditions with ducting and very dry air. - -**Key insight from link data:** On our 2.8 km terrestrial link, we measured 3-5 dB of daily signal variation driven almost entirely by humidity changes. Even at this short range, gaseous absorption is measurable and matters. - -### 75 GHz (4 mm) - -**Rare but viable.** Only 81 QSOs with distance data, but includes a 289 km contact (California marine duct). - -**Best conditions:** Very dry air (below 5 g/m³), no precipitation whatsoever, strong elevated inversions, winter months. - -**Key insight:** Rain attenuation is brutal — about 1 dB/km in light rain. Even mist or drizzle meaningfully degrades the path. - -### 122 GHz (2.5 mm) - -**The oxygen wall.** 11 QSOs, longest 139 km (California, February). - -**What makes it special:** The 118.75 GHz oxygen absorption line creates a floor of about 0.8 dB/km that you can never get below. Cold temperatures slightly narrow this absorption line, giving a small benefit in winter. - -**Practical limits:** About 30 km for reliable work, 50-80 km under good conditions, and 140 km is probably close to the theoretical tropospheric limit. - -### 134 GHz - -**The mini-window.** Only 3 QSOs in our data, longest 157 km (Germany, March). - -**What makes it special:** Sits between the O₂ 118 GHz line and the H₂O 183 GHz line — a small atmospheric window. Better than 122 GHz because you're farther from the oxygen peak, but the approaching water vapor line at 183 GHz starts to affect things. - -### 241 GHz (1.2 mm) - -**The frontier.** 5 QSOs, longest 114 km (Virginia, January). - -**What makes it special:** Between two water vapor absorption lines (183 and 325 GHz). Water vapor absorption is devastating — about 0.3 dB per kilometer per gram of moisture. At a typical 7.5 g/m³, that's 2.3 dB/km of water vapor absorption alone. - -**Best conditions:** Extremely dry air (below 3 g/m³ — essentially winter-only in most US locations), high altitude stations, short paths. - -**Practical limits:** 10-50 km for typical work. The algorithm honestly reports "path viable" or "path not viable" for this band rather than pretending to give a fine-grained score. +**Where it matters — beamwidth vs frequency:** +At 10 GHz a typical 60cm dish has ~3° beamwidth, making elevation angle errors forgiving. At 24 GHz beamwidth shrinks to ~1.5°, at 47 GHz to <1°. A 0.3° aim error that is irrelevant at 10 GHz becomes a contact killer at 47 GHz. If station profiles (antenna height, dish size, elevation setting) are added in the future, a frequency-dependent **beamwidth coupling penalty** in `margin_to_success` would be the right integration point — penalizing paths where the required aim angle to the detected duct layer exceeds the antenna's half-power beamwidth. --- -## Things We Learned From Real Data That Changed the Algorithm +## Part 8: Short-Term Prediction Model -The original algorithm (algo1) was built from theory and a handful of calibration contacts. After analyzing 58,000 QSOs and 18,000 terrestrial link measurements, several things changed: +### Approach -### 1. Wind was way overweighted +Extrapolate current conditions forward 1-6 hours using observed trends, diurnal models, and forecast data when available. -The old algorithm gave wind 18% of the total score, making it the second most important factor. The data shows it barely matters — calm winds and moderate winds produce essentially the same average contact distances. We dropped wind to 8%. +### Prediction Confidence -### 2. Humidity needed to be frequency-dependent +Based on commercial link signal prediction accuracy: -The old algorithm penalized humidity for all bands. That's correct for 24 GHz and up, but backwards for 10 GHz. The new algorithm treats humidity as beneficial at 10 GHz and harmful above that. +| Horizon | Observed Accuracy | Confidence | +|---------|-------------------|------------| +| Current | +/- 1 dB | 95% | +| +30 min | +/- 1.5 dB | 90% | +| +1 hr | +/- 2 dB | 85% | +| +2 hr | +/- 3 dB | 75% | +| +3 hr | +/- 4 dB | 60% | +| +6 hr | +/- 5 dB | 40% | -### 3. Low pressure can be good +### Diurnal Temperature Model -The old algorithm assumed high/rising pressure = stable = good propagation. The data shows the opposite at 10 GHz — low pressure systems produce the longest average distances because frontal boundaries create strong refractive gradients. The new algorithm cares about pressure change (gradient) rather than the absolute value. +```elixir +def project_temperature(current_temp_f, trend_per_hour, hours_ahead, + future_local_hour, month) do + sunrise = Enum.at(@sunrise_table, month - 1) -### 4. Binary duct detection is nearly useless + diurnal_rate = cond do + future_local_hour < sunrise - 1 -> -0.5 # Pre-dawn: slow cooling + future_local_hour < sunrise + 2 -> 0.0 # Sunrise transition + future_local_hour < 15 -> 2.0 # Morning: warming + future_local_hour < 18 -> 0.5 # Late afternoon + future_local_hour < 21 -> -1.5 # Evening: cooling + true -> -1.0 # Night: slow cooling + end -Soundings showing "duct detected" vs "no duct detected" showed almost no correlation with contact distance. This is because soundings are point samples taken twice a day, while conditions change continuously. The characteristics of the duct (height, strength, depth) matter more than whether one exists. + # Blend: current trend dominates short-term, diurnal model dominates long-term + weight = min(1.0, hours_ahead / 4.0) + blended_rate = trend_per_hour * (1.0 - weight) + diurnal_rate * weight + current_temp_f + blended_rate * hours_ahead +end +``` -### 5. Boundary layer depth matters +### Prediction Flow -A moderate boundary layer depth of 500-1000 meters is the sweet spot across all bands. This corresponds to the morning condition where the nocturnal inversion is lifting but hasn't fully broken — an elevated duct that's high enough to trap signals. +```elixir +def predict_scores(current_obs, obs_3hr_ago, forecast, band_config) do + temp_trend = (current_obs.temp_f - obs_3hr_ago.temp_f) / 3 + dp_trend = (current_obs.dewpoint_f - obs_3hr_ago.dewpoint_f) / 3 + pressure_trend = (current_obs.slp - obs_3hr_ago.slp) / 3 -### 6. Short-path physics are inverted + for hours_ahead <- 1..6 do + future_time = DateTime.add(current_obs.observed_at, hours_ahead * 3600) + month = future_time.month -Our terrestrial link data showed that sub-refractive conditions (less bending than normal) actually improve signal on short, line-of-sight paths. This is the opposite of what helps long-range ham contacts. The algorithm distinguishes between LOS paths and beyond-LOS paths for this reason. + projected_temp = project_temperature(current_obs.temp_f, temp_trend, + hours_ahead, future_time.hour, month) + projected_dp = current_obs.dewpoint_f + dp_trend * hours_ahead + projected_slp = current_obs.slp + pressure_trend * hours_ahead + projected_sky = forecast_value(forecast, :sky, hours_ahead) || current_obs.sky_condition + projected_rain = forecast_value(forecast, :rain_rate, hours_ahead) || 0 + projected_wind = forecast_value(forecast, :wind_kts, hours_ahead) || current_obs.wind_speed_kts + + # Compute absolute humidity from projected values + tc = (projected_temp - 32) * 5 / 9 + td_c = (projected_dp - 32) * 5 / 9 + es = 6.112 * :math.exp(17.67 * tc / (tc + 243.5)) + ed = 6.112 * :math.exp(17.67 * td_c / (td_c + 243.5)) + rh = min(100, ed / es * 100) + abs_hum = 217 * (rh / 100) * es / (tc + 273.15) + + factors = %{ + humidity: score_humidity(abs_hum, band_config), + wind: score_wind(projected_wind), + sky: score_sky(projected_sky), + time_of_day: score_time_of_day(future_time.hour, future_time.minute, month) |> elem(0), + td_depression: score_td_depression(projected_temp, projected_dp, band_config), + season: score_season(month, band_config), + pressure: score_pressure(projected_slp, current_obs.slp), + rain: score_rain(projected_rain, band_config), + refractivity: 50 # Cannot predict from surface obs alone + } + + %{ + hours_ahead: hours_ahead, + time: future_time, + score: composite_score(factors), + factors: factors, + confidence: prediction_confidence(hours_ahead) + } + end +end + +def prediction_confidence(hours_ahead) do + case hours_ahead do + 1 -> 0.85 + 2 -> 0.75 + 3 -> 0.60 + 4 -> 0.50 + 5 -> 0.40 + 6 -> 0.30 + _ -> 0.20 + end +end +``` --- -## What the Algorithm Can't Do +## Part 9: Sounding & Refractivity Analysis -- **It can't predict the unpredictable.** A random thunderstorm cell, a sudden wind shift, or an unexpected frontal passage will invalidate the forecast. -- **It can't see between soundings.** Upper-air data is only available twice a day. The atmosphere between 00Z and 12Z soundings is interpolated, not measured. -- **It doesn't know your terrain.** Two stations on mountaintops may have line-of-sight at 200 km without needing any atmospheric help. Two stations in flat terrain need ducting for the same distance. Without SRTM elevation data for each path, the algorithm assumes all paths need atmospheric bending. -- **It has ±2-3 dB of inherent noise.** Even with perfect weather data, atmospheric scintillation, equipment thermal effects, and multipath geometry mean the real signal will bounce around by a few dB regardless of what the algorithm predicts. -- **Sparse data above 75 GHz.** The 122/134/241 GHz band parameters are based on physics models and a handful of contacts, not the robust statistical analysis we have for 10 and 24 GHz. +### Refractivity Profile from Sounding + +```elixir +def compute_refractivity_profile(levels, sfc_height_m) do + Enum.map(levels, fn level -> + t_k = level.temp_c + 273.15 + e = 6.1121 * :math.exp((18.678 - level.temp_c / 234.5) * (level.temp_c / (257.14 + level.temp_c))) + e_actual = if level.dewpoint_c, do: 6.1121 * :math.exp((18.678 - level.dewpoint_c / 234.5) * (level.dewpoint_c / (257.14 + level.dewpoint_c))), else: 0 + + n = 77.6 * level.pressure_hpa / t_k + 3.73e5 * e_actual / (t_k * t_k) + h_agl = level.height_m - sfc_height_m + m = n + 0.157 * h_agl + + %{height_agl: h_agl, n: n, m: m, temp_c: level.temp_c, dewpoint_c: level.dewpoint_c} + end) +end +``` + +### Duct Detection + +Duct exists where dM/dh < 0. Filter for strength > 2 M-units. + +```elixir +def detect_ducts(profile) do + profile + |> Enum.chunk_every(2, 1, :discard) + |> Enum.reduce({[], nil}, fn [below, above], {ducts, duct_start} -> + dm = above.m - below.m + + cond do + dm < 0 and duct_start == nil -> + {ducts, %{base: below.height_agl, base_m: below.m}} + + dm >= 0 and duct_start != nil -> + strength = duct_start.base_m - below.m + if strength > 2 do + duct = %{base: duct_start.base, top: below.height_agl, strength: strength} + {[duct | ducts], nil} + else + {ducts, nil} + end + + true -> + {ducts, duct_start} + end + end) + |> elem(0) + |> Enum.reverse() +end +``` + +### Inversion Detection + +Temperature increasing with height. Merge adjacent inversions within 200m gap. Filter: strength >= 0.5C, base < 5000m AGL. + +### Stability Indices + +``` +K-Index = (T850 - T500) + Td850 - (T700 - Td700) + +Lifted Index = T500 - (Tsfc - (h500 - h_sfc) * 0.00976) + LI < 0: Unstable (convection likely, inversion destroyed) + LI > 0: Stable (inversion maintained) + +Precipitable Water = sum[(MR_i + MR_{i-1}) / 2 * dP / (9.81 * 10)] + MR = 622 * e / (P - e) +``` + +### Boundary Layer Depth + +Find height where potential temperature (theta = T + 9.8 * h/1000) exceeds surface theta by 2C. The 500-1000m sweet spot indicates an elevated inversion — high enough to trap signals, not so deep that full mixing has occurred. --- -## Data Sources +## Part 10: Band-Specific Propagation Mechanisms -- **58,282 QSOs** from ARRL microwave contest logs (2019-2024), cross-referenced with weather -- **95 ASOS surface observation stations** providing temperature, dewpoint, wind, pressure, sky condition -- **9 RAOB sounding stations** providing vertical atmospheric profiles twice daily -- **18,540 signal measurements** from 7 terrestrial microwave links at 11, 24, and 68 GHz (March 2026) -- **ITU-R Recommendations** P.453 (refractivity), P.525 (free-space loss), P.676 (gaseous absorption), P.838 (rain attenuation) +### Coupling Sensitivity by Frequency + +Duct coupling geometry becomes increasingly critical at higher frequencies due to narrower antenna beamwidths. A dish aimed 0.3° away from the optimal duct grazing angle: +- **10 GHz** (~3° beamwidth): Still within half-power beam — negligible loss +- **24 GHz** (~1.5° beamwidth): Approaching beam edge — moderate coupling loss +- **47 GHz** (<1° beamwidth): Outside half-power beam — potential contact killer +- **75+ GHz** (<0.5° beamwidth): Precision aim required — elevation error dominates + +For surface ducts, the beam must arrive at <0.5° grazing incidence to be trapped. For elevated ducts (500-1500m AGL), the optimal elevation angle is path-distance dependent: slightly positive at close range, near-zero at the "sweet spot" distance, and slightly negative at extreme range due to Earth curvature. + +### 10 GHz (3cm) — Tropospheric Ducting Band + +**Primary mechanisms:** Ducting, enhanced refraction +**Key variable:** Refractivity profile, NOT humidity absorption (0.012 dB/km total is negligible) +**Best conditions:** Moderate-high humidity (12-20 g/m^3), temperature inversions, stable atmosphere, late evening through early morning +**Unique:** Largely insensitive to rain. Can propagate through cloud decks. Marine ducting produces 1000+ km coastal paths. Frontal boundaries create strong refractive gradients. + +### 24 GHz (1.2cm) — Water Vapor Line Band + +**Primary mechanisms:** Ducting (reduced by absorption), enhanced refraction +**Key variable:** Absolute humidity (22.235 GHz H2O line makes this THE most humidity-sensitive band) +**Best conditions:** Very dry air (<8 g/m^3), cold season (Nov-Mar), clear skies, pre-dawn through early morning +**Unique:** 10x more sensitive to water vapor than 10 GHz. Summer Gulf moisture devastates range. Rain scatter is a viable alternative mechanism (710 km QSO documented). + +### 47 GHz (6mm) — Atmospheric Window + +**Primary mechanisms:** Ducting (in atmospheric window), enhanced LOS +**Key variable:** Balance of humidity and refractivity; very dry air dramatically helps +**Best conditions:** Dry air (<8 g/m^3), clear skies, strong inversions, early morning +**Unique:** Window between 22 GHz H2O and 60 GHz O2. O2 absorption ~0.045 dB/km is fixed. Ducting is the ONLY way beyond ~150 km. + +### 68 GHz — V-Band Edge + +**Primary mechanisms:** LOS only (O2 absorption limits range) +**Key variable:** O2 wing absorption (~0.9 dB/km, weather-independent) + humidity +**Best conditions:** Cold/dry air, no precipitation, short paths +**Unique:** Validated by link data showing 3-5 dB diurnal fades on 2.8 km path. O2 absorption caps practical range regardless of conditions. Viable for short links (<5 km), very challenging for beyond-LOS. + +### 75 GHz (4mm) — Window Band + +**Primary mechanisms:** Rare ducting, enhanced LOS +**Key variable:** Dry air + no precipitation +**Best conditions:** Very dry (<5 g/m^3), no rain, strong inversions, winter +**Unique:** 289 km record (California marine duct). Only 81 QSOs with distance data. Rain attenuation severe (~1 dB/km at 4 mm/hr). + +### 122 GHz (2.5mm) — O2 Line Wing + +**Primary mechanisms:** Enhanced LOS, rare ducting +**Key variable:** O2 absorption from 118.75 GHz line (~0.8 dB/km, cannot be improved by weather) +**Best conditions:** Cold temperatures (reduce O2 line broadening), very dry, no rain +**Unique:** 139 km record (California, February). Practically limited to ~50 km reliable paths. + +### 134 GHz — Mini Window + +**Primary mechanisms:** Enhanced LOS +**Key variable:** Between O2 118 and H2O 183 lines +**Best conditions:** Cold, dry, no precipitation +**Unique:** 157 km record (Germany, March). Better than 122 GHz due to distance from O2 line. + +### 241 GHz (1.2mm) — Submillimeter + +**Primary mechanisms:** LOS only +**Key variable:** H2O absorption dominates (~0.3 dB/km per g/m^3) +**Best conditions:** Extremely dry (<3 g/m^3), winter-only in most US locations, high altitude stations +**Unique:** 114 km record (Virginia, January). Total path loss at 100 km is ~410 dB without ducting. Realistic to display "viable / not viable" rather than a score. + +--- + +## Part 11: Data Flow & Implementation + +``` +Surface Observations (ASOS, every 5-20 min) + -> temp, dewpoint, wind, pressure, sky, visibility + -> compute: abs_humidity, Td depression + -> per-band scoring functions + -> composite score per band + -> 6-hour prediction timeline + +Sounding Data (RAOB 00Z/12Z) + HRRR Model (hourly) + -> refractivity profile, dN/dh gradient, ducts, BL depth + -> refractivity score component + -> regime classification (LOS vs beyond-LOS for specific paths) + +Terrain Data (SRTM) + -> path profile, Fresnel clearance, earth bulge + -> determines LOS vs beyond-LOS regime + -> diffraction loss calculation + +Link Budget (point-to-point) + -> FSPL + gaseous + rain + diffraction - duct enhancement + -> margin = RX power - sensitivity + -> success % = margin_to_success(margin, prop_score) +``` + +### Display: Band Conditions Panel + +For each band: +- Current score (0-100, colored badge) +- Estimated range (km, from score tier table) +- Key limiting factor ("High humidity: 16 g/m^3", "Strong inversion detected") +- Trend arrow (improving/stable/degrading from last hour) +- 6-hour prediction timeline with confidence shading + +--- + +## Constants Reference + +| Constant | Value | Source | +|----------|-------|--------| +| Earth radius | 6371 km | WGS-84 mean | +| Standard K-factor | 4/3 | Standard atmosphere | +| Standard surface N | 315 | ITU-R P.453 | +| Standard dN/dh | -40 /km | ITU-R P.453 | +| Humidity penalty 24 GHz | 1.6 | Near 22.235 GHz H2O peak | +| Humidity penalty 47 GHz | 1.0 | Atmospheric window | +| Humidity penalty 68 GHz | 1.4 | 60 GHz O2 wing + H2O | +| Humidity penalty 241 GHz | 3.0 | Between H2O 183 & 325 | +| Duct M-unit threshold | 2 | Noise filter | +| Inversion min strength | 0.5C | Below is noise | +| Inversion height limit | 5000m AGL | Above irrelevant | +| BL depth sweet spot | 500-1000m | Empirical from QSO data | +| Signal prediction floor | +/- 2-3 dB | Measured from link data | + +## ITU-R References + +- P.453-14: Radio refractivity +- P.525: Free-space path loss +- P.676-13: Gaseous absorption (O2 + H2O) +- P.838-3: Specific rain attenuation +- P.526: Diffraction +- P.452: Interference between stations + +## Data References + +- ARRL Microwave Contest QSO database: 57,492 contacts with distance (2019-2024) +- IEM ASOS observations: 95 stations, +/-2hr window match +- IEM RAOB soundings: 9 stations, +/-6hr window match +- Commercial link data: 18,540 samples, 7 links at 11/24/68 GHz (March 2026) +- HRRR model: hourly refractivity profiles (March 2026 validation period) diff --git a/config/config.exs b/config/config.exs index 8285b654..62ca5075 100644 --- a/config/config.exs +++ b/config/config.exs @@ -44,14 +44,15 @@ config :microwaveprop, MicrowavepropWeb.Endpoint, config :microwaveprop, Oban, repo: Microwaveprop.Repo, - queues: [solar: 1, weather: 3, enqueue: 1, hrrr: 20, terrain: 4], + queues: [solar: 1, weather: 3, enqueue: 1, hrrr: 20, terrain: 4, commercial: 2], plugins: [ {Oban.Plugins.Pruner, max_age: 3600 * 24}, {Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 30)}, {Oban.Plugins.Cron, crontab: [ {"0 8 * * *", Microwaveprop.Workers.SolarIndexWorker}, - {"0 */4 * * *", Microwaveprop.Workers.QsoWeatherEnqueueWorker} + {"0 */4 * * *", Microwaveprop.Workers.QsoWeatherEnqueueWorker}, + {"*/5 * * * *", Microwaveprop.Commercial.PollWorker} ]} ] diff --git a/docs/plans/2026-03-30-commercial-link-monitoring.md b/docs/plans/2026-03-30-commercial-link-monitoring.md new file mode 100644 index 00000000..feebb0ae --- /dev/null +++ b/docs/plans/2026-03-30-commercial-link-monitoring.md @@ -0,0 +1,1255 @@ +# Commercial Link Monitoring Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Poll UBNT AirFiber radios via SNMP every 5 minutes, store signal metrics in PostgreSQL, and fetch ASOS weather alongside each cycle for propagation correlation. + +**Architecture:** New `Commercial` context with `Link` and `Sample` schemas. An Oban cron worker (`PollWorker`) runs every 5 minutes, queries all enabled links, shells out to `snmpget`/`snmpgetnext` (net-snmp CLI) to poll each radio, inserts signal samples, then fetches ASOS weather for each unique weather station via the existing `IemClient`. Two radio types are supported: AF11X (single `snmpget`) and AF60-LR (`snmpget` + `snmpgetnext` for MAC-indexed station table). + +**Tech Stack:** Elixir/Phoenix, Ecto, Oban, net-snmp CLI (`snmpget`/`snmpgetnext`), existing `Weather.IemClient` + +--- + +## Task 1: Link Schema + Migration + +**Files:** +- Create: `lib/microwaveprop/commercial/link.ex` +- Create: `priv/repo/migrations/TIMESTAMP_create_commercial_links.exs` +- Create: `test/microwaveprop/commercial/link_test.exs` + +**Step 1: Write the failing test** + +```elixir +# test/microwaveprop/commercial/link_test.exs +defmodule Microwaveprop.Commercial.LinkTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial.Link + + @valid_attrs %{ + label: "verona-to-climax", + host: "10.250.1.26", + community: "public", + radio_type: "af11x", + endpoint_a: %{lat: 33.246171, lon: -96.426516, alt_m: 205.4}, + endpoint_b: %{lat: 33.187294, lon: -96.448128, alt_m: 191.0}, + weather_station: "KTKI", + enabled: true + } + + describe "changeset/2" do + test "valid attributes" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert changeset.valid? + end + + test "requires label, host, community, radio_type" do + changeset = Link.changeset(%Link{}, %{}) + + assert %{ + label: ["can't be blank"], + host: ["can't be blank"], + community: ["can't be blank"], + radio_type: ["can't be blank"] + } = errors_on(changeset) + end + + test "validates radio_type inclusion" do + changeset = Link.changeset(%Link{}, %{@valid_attrs | radio_type: "unknown"}) + assert %{radio_type: ["is invalid"]} = errors_on(changeset) + end + + test "defaults enabled to true" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert Ecto.Changeset.get_field(changeset, :enabled) == true + end + + test "persists to the database" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert {:ok, link} = Repo.insert(changeset) + assert link.label == "verona-to-climax" + assert link.host == "10.250.1.26" + assert link.radio_type == "af11x" + assert link.endpoint_a["lat"] == 33.246171 + end + + test "enforces unique label" do + assert {:ok, _} = Repo.insert(Link.changeset(%Link{}, @valid_attrs)) + assert {:error, changeset} = Repo.insert(Link.changeset(%Link{}, @valid_attrs)) + assert %{label: ["has already been taken"]} = errors_on(changeset) + end + end +end +``` + +**Step 2: Run test to verify it fails** + +Run: `mix test test/microwaveprop/commercial/link_test.exs` +Expected: Compilation error — `Microwaveprop.Commercial.Link` not found + +**Step 3: Generate the migration** + +Run: `mix ecto.gen.migration create_commercial_links` + +Then fill in the migration: + +```elixir +defmodule Microwaveprop.Repo.Migrations.CreateCommercialLinks do + use Ecto.Migration + + def change do + create table(:commercial_links, primary_key: false) do + add :id, :binary_id, primary_key: true + add :label, :string, null: false + add :host, :string, null: false + add :community, :string, null: false + add :radio_type, :string, null: false, default: "af11x" + add :endpoint_a, :map + add :endpoint_b, :map + add :weather_station, :string + add :enabled, :boolean, null: false, default: true + + timestamps(type: :utc_datetime) + end + + create unique_index(:commercial_links, [:label]) + create index(:commercial_links, [:enabled]) + end +end +``` + +**Step 4: Write the schema** + +```elixir +# lib/microwaveprop/commercial/link.ex +defmodule Microwaveprop.Commercial.Link do + @moduledoc false + use Ecto.Schema + + import Ecto.Changeset + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + + schema "commercial_links" do + field :label, :string + field :host, :string + field :community, :string + field :radio_type, :string, default: "af11x" + field :endpoint_a, :map + field :endpoint_b, :map + field :weather_station, :string + field :enabled, :boolean, default: true + + timestamps(type: :utc_datetime) + end + + @required_fields ~w(label host community radio_type)a + @optional_fields ~w(endpoint_a endpoint_b weather_station enabled)a + + def changeset(link, attrs) do + link + |> cast(attrs, @required_fields ++ @optional_fields) + |> validate_required(@required_fields) + |> validate_inclusion(:radio_type, ~w(af11x af60)) + |> unique_constraint(:label) + end +end +``` + +**Step 5: Run migration and tests** + +Run: `mix ecto.migrate && mix test test/microwaveprop/commercial/link_test.exs` +Expected: All tests pass + +**Step 6: Commit** + +``` +feat: add commercial link schema and migration +``` + +--- + +## Task 2: Seed Migration — Import Links from links.yaml + +**Files:** +- Create: `priv/repo/migrations/TIMESTAMP_seed_commercial_links.exs` +- No test file — data migration verified by running `mix ecto.migrate` + +**Step 1: Write the seed migration** + +Run: `mix ecto.gen.migration seed_commercial_links` + +```elixir +defmodule Microwaveprop.Repo.Migrations.SeedCommercialLinks do + use Ecto.Migration + + def up do + now = DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_naive() + + links = [ + %{ + label: "verona-to-climax", + host: "10.250.1.26", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.246171,"lon":-96.426516,"alt_m":205.4}|, + endpoint_b: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + weather_station: "KTKI" + }, + %{ + label: "new-hope-to-core", + host: "10.250.1.58", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.207279,"lon":-96.537762,"alt_m":216.4}|, + endpoint_b: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + weather_station: "KTKI" + }, + %{ + label: "core-new-hope", + host: "10.250.1.61", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + endpoint_b: ~s|{"lat":33.207279,"lon":-96.537762,"alt_m":216.4}|, + weather_station: "KTKI" + }, + %{ + label: "982_380_60LR", + host: "10.250.1.34", + community: "kdyyJrT0Mm", + radio_type: "af60", + endpoint_a: ~s|{"lat":33.148921,"lon":-96.4958,"alt_m":186.5}|, + endpoint_b: ~s|{"lat":33.174135,"lon":-96.491935,"alt_m":200.4}|, + weather_station: "KTKI" + }, + %{ + label: "380_982_60LR", + host: "10.250.1.37", + community: "kdyyJrT0Mm", + radio_type: "af60", + endpoint_a: ~s|{"lat":33.174135,"lon":-96.491935,"alt_m":200.4}|, + endpoint_b: ~s|{"lat":33.148921,"lon":-96.4958,"alt_m":186.5}|, + weather_station: "KTKI" + }, + %{ + label: "core-to-climax", + host: "10.250.1.93", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + endpoint_b: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + weather_station: "KTKI" + }, + %{ + label: "climax-to-core", + host: "10.250.1.90", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + endpoint_b: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + weather_station: "KTKI" + } + ] + + for link <- links do + execute(""" + INSERT INTO commercial_links (id, label, host, community, radio_type, endpoint_a, endpoint_b, weather_station, enabled, inserted_at, updated_at) + VALUES (gen_random_uuid(), '#{link.label}', '#{link.host}', '#{link.community}', '#{link.radio_type}', '#{link.endpoint_a}'::jsonb, '#{link.endpoint_b}'::jsonb, '#{link.weather_station}', true, '#{now}', '#{now}') + ON CONFLICT (label) DO NOTHING + """) + end + end + + def down do + execute("DELETE FROM commercial_links") + end +end +``` + +**Step 2: Run migration** + +Run: `mix ecto.migrate` +Expected: 7 rows inserted into `commercial_links` + +**Step 3: Verify in iex** + +Run: `mix run -e "Microwaveprop.Repo.all(Microwaveprop.Commercial.Link) |> length() |> IO.inspect()"` +Expected: `7` + +**Step 4: Commit** + +``` +feat: seed commercial links from existing link definitions +``` + +--- + +## Task 3: Sample Schema + Migration + +**Files:** +- Create: `lib/microwaveprop/commercial/sample.ex` +- Create: `priv/repo/migrations/TIMESTAMP_create_commercial_samples.exs` +- Create: `test/microwaveprop/commercial/sample_test.exs` + +**Step 1: Write the failing test** + +```elixir +# test/microwaveprop/commercial/sample_test.exs +defmodule Microwaveprop.Commercial.SampleTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial.Link + alias Microwaveprop.Commercial.Sample + + defp create_link do + %Link{} + |> Link.changeset(%{ + label: "test-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x" + }) + |> Repo.insert!() + end + + @sample_attrs %{ + sampled_at: ~U[2026-03-30 12:00:00.000000Z], + rx_power_0: -55.0, + tx_power: 20.0, + link_state: 1, + link_uptime: 86400, + cur_tx_mod_rate: 6, + rx_capacity: 450, + tx_capacity: 450 + } + + describe "changeset/2" do + test "valid attributes with link_id" do + link = create_link() + attrs = Map.put(@sample_attrs, :link_id, link.id) + changeset = Sample.changeset(%Sample{}, attrs) + assert changeset.valid? + end + + test "requires link_id and sampled_at" do + changeset = Sample.changeset(%Sample{}, %{}) + assert %{link_id: ["can't be blank"], sampled_at: ["can't be blank"]} = errors_on(changeset) + end + + test "persists to the database" do + link = create_link() + attrs = Map.put(@sample_attrs, :link_id, link.id) + + assert {:ok, sample} = Repo.insert(Sample.changeset(%Sample{}, attrs)) + assert sample.rx_power_0 == -55.0 + assert sample.link_id == link.id + end + + test "all signal fields are optional" do + link = create_link() + attrs = %{link_id: link.id, sampled_at: ~U[2026-03-30 12:00:00.000000Z]} + changeset = Sample.changeset(%Sample{}, attrs) + assert changeset.valid? + end + end +end +``` + +**Step 2: Run test to verify it fails** + +Run: `mix test test/microwaveprop/commercial/sample_test.exs` +Expected: Compilation error — `Microwaveprop.Commercial.Sample` not found + +**Step 3: Generate migration** + +Run: `mix ecto.gen.migration create_commercial_samples` + +```elixir +defmodule Microwaveprop.Repo.Migrations.CreateCommercialSamples do + use Ecto.Migration + + def change do + create table(:commercial_samples, primary_key: false) do + add :id, :binary_id, primary_key: true + add :link_id, references(:commercial_links, type: :binary_id, on_delete: :delete_all), null: false + add :sampled_at, :utc_datetime_usec, null: false + + # Local radio + add :rx_power_0, :float + add :rx_power_1, :float + add :tx_power, :float + add :tx_freq_mhz, :integer + add :rx_freq_mhz, :integer + add :radio_temp_0_c, :float + add :radio_temp_1_c, :float + + # Modulation & capacity + add :cur_tx_mod_rate, :integer + add :remote_tx_mod_rate, :integer + add :rx_capacity, :integer + add :tx_capacity, :integer + + # Remote radio + add :remote_tx_power, :float + add :remote_rx_power_0, :float + add :remote_rx_power_1, :float + + # Link status + add :link_state, :integer + add :link_uptime, :integer + add :link_distance_m, :integer + + timestamps(type: :utc_datetime) + end + + create index(:commercial_samples, [:link_id, :sampled_at]) + create index(:commercial_samples, [:sampled_at]) + end +end +``` + +**Step 4: Write the schema** + +```elixir +# lib/microwaveprop/commercial/sample.ex +defmodule Microwaveprop.Commercial.Sample do + @moduledoc false + use Ecto.Schema + + import Ecto.Changeset + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + + schema "commercial_samples" do + belongs_to :link, Microwaveprop.Commercial.Link + field :sampled_at, :utc_datetime_usec + + # Local radio + field :rx_power_0, :float + field :rx_power_1, :float + field :tx_power, :float + field :tx_freq_mhz, :integer + field :rx_freq_mhz, :integer + field :radio_temp_0_c, :float + field :radio_temp_1_c, :float + + # Modulation & capacity + field :cur_tx_mod_rate, :integer + field :remote_tx_mod_rate, :integer + field :rx_capacity, :integer + field :tx_capacity, :integer + + # Remote radio + field :remote_tx_power, :float + field :remote_rx_power_0, :float + field :remote_rx_power_1, :float + + # Link status + field :link_state, :integer + field :link_uptime, :integer + field :link_distance_m, :integer + + timestamps(type: :utc_datetime) + end + + @required_fields ~w(link_id sampled_at)a + @optional_fields ~w(rx_power_0 rx_power_1 tx_power tx_freq_mhz rx_freq_mhz + radio_temp_0_c radio_temp_1_c cur_tx_mod_rate remote_tx_mod_rate + rx_capacity tx_capacity remote_tx_power remote_rx_power_0 remote_rx_power_1 + link_state link_uptime link_distance_m)a + + def changeset(sample, attrs) do + sample + |> cast(attrs, @required_fields ++ @optional_fields) + |> validate_required(@required_fields) + |> foreign_key_constraint(:link_id) + end +end +``` + +**Step 5: Run migration and tests** + +Run: `mix ecto.migrate && mix test test/microwaveprop/commercial/sample_test.exs` +Expected: All tests pass + +**Step 6: Commit** + +``` +feat: add commercial sample schema and migration +``` + +--- + +## Task 4: Commercial Context Module + +**Files:** +- Create: `lib/microwaveprop/commercial.ex` +- Create: `test/microwaveprop/commercial_test.exs` + +**Step 1: Write the failing test** + +```elixir +# test/microwaveprop/commercial_test.exs +defmodule Microwaveprop.CommercialTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial + + @link_attrs %{ + label: "test-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x", + weather_station: "KTKI", + enabled: true + } + + describe "enabled_links/0" do + test "returns only enabled links" do + {:ok, _enabled} = Commercial.create_link(@link_attrs) + {:ok, _disabled} = Commercial.create_link(%{@link_attrs | label: "disabled-link", enabled: false}) + + links = Commercial.enabled_links() + assert length(links) == 1 + assert hd(links).label == "test-link" + end + + test "returns empty list when no enabled links" do + assert Commercial.enabled_links() == [] + end + end + + describe "create_link/1" do + test "creates a link with valid attrs" do + assert {:ok, link} = Commercial.create_link(@link_attrs) + assert link.label == "test-link" + assert link.enabled == true + end + + test "returns error for invalid attrs" do + assert {:error, _changeset} = Commercial.create_link(%{}) + end + end + + describe "create_sample/1" do + test "creates a sample for a link" do + {:ok, link} = Commercial.create_link(@link_attrs) + + attrs = %{ + link_id: link.id, + sampled_at: DateTime.utc_now(), + rx_power_0: -55.0, + tx_power: 20.0, + link_state: 1 + } + + assert {:ok, sample} = Commercial.create_sample(attrs) + assert sample.rx_power_0 == -55.0 + assert sample.link_id == link.id + end + end + + describe "weather_stations/0" do + test "returns unique weather stations from enabled links" do + {:ok, _} = Commercial.create_link(@link_attrs) + {:ok, _} = Commercial.create_link(%{@link_attrs | label: "link-2", host: "10.0.0.2"}) + {:ok, _} = Commercial.create_link(%{@link_attrs | label: "link-3", host: "10.0.0.3", weather_station: "KDFW"}) + + stations = Commercial.weather_stations() + assert Enum.sort(stations) == ["KDFW", "KTKI"] + end + + test "excludes disabled links" do + {:ok, _} = Commercial.create_link(%{@link_attrs | enabled: false}) + + assert Commercial.weather_stations() == [] + end + + test "excludes nil weather stations" do + {:ok, _} = Commercial.create_link(%{@link_attrs | weather_station: nil}) + + assert Commercial.weather_stations() == [] + end + end +end +``` + +**Step 2: Run test to verify it fails** + +Run: `mix test test/microwaveprop/commercial_test.exs` +Expected: Compilation error — `Microwaveprop.Commercial` not found + +**Step 3: Write the context module** + +```elixir +# lib/microwaveprop/commercial.ex +defmodule Microwaveprop.Commercial do + @moduledoc false + + import Ecto.Query + + alias Microwaveprop.Commercial.Link + alias Microwaveprop.Commercial.Sample + alias Microwaveprop.Repo + + def enabled_links do + Link + |> where([l], l.enabled == true) + |> Repo.all() + end + + def create_link(attrs) do + %Link{} + |> Link.changeset(attrs) + |> Repo.insert() + end + + def create_sample(attrs) do + %Sample{} + |> Sample.changeset(attrs) + |> Repo.insert() + end + + def weather_stations do + Link + |> where([l], l.enabled == true and not is_nil(l.weather_station)) + |> select([l], l.weather_station) + |> distinct(true) + |> Repo.all() + end +end +``` + +**Step 4: Run tests** + +Run: `mix test test/microwaveprop/commercial_test.exs` +Expected: All tests pass + +**Step 5: Commit** + +``` +feat: add commercial context with link and sample CRUD +``` + +--- + +## Task 5: SNMP Client + +**Files:** +- Create: `lib/microwaveprop/commercial/snmp_client.ex` +- Create: `test/microwaveprop/commercial/snmp_client_test.exs` + +**Step 1: Write the failing test** + +The SNMP client shells out to `snmpget`/`snmpgetnext`. Tests will mock `System.cmd/3` by making the module accept a command runner function (dependency injection via application config in test). + +```elixir +# test/microwaveprop/commercial/snmp_client_test.exs +defmodule Microwaveprop.Commercial.SnmpClientTest do + use ExUnit.Case, async: true + + alias Microwaveprop.Commercial.SnmpClient + + @af11x_output """ + iso.3.6.1.4.1.41112.1.3.2.1.11.1 = INTEGER: -55 + iso.3.6.1.4.1.41112.1.3.2.1.14.1 = INTEGER: -56 + iso.3.6.1.4.1.41112.1.3.1.1.9.1 = INTEGER: 20 + iso.3.6.1.4.1.41112.1.3.2.1.2.1 = INTEGER: 6 + iso.3.6.1.4.1.41112.1.3.2.1.18.1 = INTEGER: 6 + iso.3.6.1.4.1.41112.1.3.2.1.5.1 = INTEGER: 450 + iso.3.6.1.4.1.41112.1.3.2.1.6.1 = INTEGER: 450 + iso.3.6.1.4.1.41112.1.3.2.1.17.1 = INTEGER: 20 + iso.3.6.1.4.1.41112.1.3.2.1.19.1 = INTEGER: -54 + iso.3.6.1.4.1.41112.1.3.2.1.22.1 = INTEGER: -57 + iso.3.6.1.4.1.41112.1.3.2.1.26.1 = INTEGER: 1 + iso.3.6.1.4.1.41112.1.3.2.1.44.1 = INTEGER: 86400 + iso.3.6.1.4.1.41112.1.3.2.1.8.1 = INTEGER: 45 + iso.3.6.1.4.1.41112.1.3.2.1.10.1 = INTEGER: 47 + iso.3.6.1.4.1.41112.1.3.1.1.5.1 = INTEGER: 11245 + iso.3.6.1.4.1.41112.1.3.1.1.6.1 = INTEGER: 10720 + iso.3.6.1.4.1.41112.1.3.2.1.4.1 = INTEGER: 7200 + """ + + describe "parse_snmpget_output/2" do + test "parses AF11X snmpget output into signal map" do + result = SnmpClient.parse_snmpget_output(@af11x_output, :af11x) + + assert result.rx_power_0 == -55 + assert result.rx_power_1 == -56 + assert result.tx_power == 20 + assert result.cur_tx_mod_rate == 6 + assert result.remote_tx_mod_rate == 6 + assert result.rx_capacity == 450 + assert result.tx_capacity == 450 + assert result.remote_tx_power == 20 + assert result.remote_rx_power_0 == -54 + assert result.remote_rx_power_1 == -57 + assert result.link_state == 1 + assert result.link_uptime == 86400 + assert result.radio_temp_0_c == 45 + assert result.radio_temp_1_c == 47 + assert result.tx_freq_mhz == 11245 + assert result.rx_freq_mhz == 10720 + assert result.link_distance_m == 7200 + end + + test "returns empty map for empty output" do + assert SnmpClient.parse_snmpget_output("", :af11x) == %{} + end + + test "skips lines with No Such Instance" do + output = """ + iso.3.6.1.4.1.41112.1.3.2.1.11.1 = INTEGER: -55 + iso.3.6.1.4.1.41112.1.3.2.1.14.1 = No Such Instance currently exists at this OID + """ + + result = SnmpClient.parse_snmpget_output(output, :af11x) + assert result.rx_power_0 == -55 + refute Map.has_key?(result, :rx_power_1) + end + end + + @af60_static_output """ + iso.3.6.1.4.1.41112.1.11.1.2.5.1 = INTEGER: 52 + iso.3.6.1.4.1.41112.1.11.1.2.6.1 = INTEGER: 1 + iso.3.6.1.4.1.41112.1.11.1.1.2.1 = INTEGER: 60480 + """ + + @af60_station_output """ + iso.3.6.1.4.1.41112.1.11.1.3.1.3.170.211.109.42.26.137 = INTEGER: -58 + iso.3.6.1.4.1.41112.1.11.1.3.1.4.170.211.109.42.26.137 = INTEGER: 18 + iso.3.6.1.4.1.41112.1.11.1.3.1.5.170.211.109.42.26.137 = INTEGER: 9 + iso.3.6.1.4.1.41112.1.11.1.3.1.6.170.211.109.42.26.137 = INTEGER: 8 + iso.3.6.1.4.1.41112.1.11.1.3.1.7.170.211.109.42.26.137 = INTEGER: 900000 + iso.3.6.1.4.1.41112.1.11.1.3.1.8.170.211.109.42.26.137 = INTEGER: 850000 + iso.3.6.1.4.1.41112.1.11.1.3.1.15.170.211.109.42.26.137 = INTEGER: 2800 + iso.3.6.1.4.1.41112.1.11.1.3.1.17.170.211.109.42.26.137 = INTEGER: 8640000 + iso.3.6.1.4.1.41112.1.11.1.3.1.18.170.211.109.42.26.137 = INTEGER: -57 + iso.3.6.1.4.1.41112.1.11.1.3.1.19.170.211.109.42.26.137 = INTEGER: 19 + """ + + describe "parse_af60_output/2" do + test "parses AF60 static + station output" do + result = SnmpClient.parse_af60_output(@af60_static_output, @af60_station_output) + + assert result.radio_temp_0_c == 52 + assert result.link_state == 1 + assert result.tx_freq_mhz == 60480 + assert result.rx_power_0 == -58 + assert result.tx_power == 18 + assert result.cur_tx_mod_rate == 9 + assert result.remote_tx_mod_rate == 8 + assert result.tx_capacity == 900 + assert result.rx_capacity == 850 + assert result.link_distance_m == 2800 + assert result.link_uptime == 86400 + assert result.remote_rx_power_0 == -57 + assert result.remote_tx_power == 19 + end + end +end +``` + +**Step 2: Run test to verify it fails** + +Run: `mix test test/microwaveprop/commercial/snmp_client_test.exs` +Expected: Compilation error — `Microwaveprop.Commercial.SnmpClient` not found + +**Step 3: Write the SNMP client** + +```elixir +# lib/microwaveprop/commercial/snmp_client.ex +defmodule Microwaveprop.Commercial.SnmpClient do + @moduledoc false + + require Logger + + # UBNT-AirFIBER-MIB OIDs (base 1.3.6.1.4.1.41112.1.3) + @af_base "1.3.6.1.4.1.41112.1.3" + @af11x_oids %{ + "#{@af_base}.2.1.11.1" => :rx_power_0, + "#{@af_base}.2.1.14.1" => :rx_power_1, + "#{@af_base}.1.1.9.1" => :tx_power, + "#{@af_base}.2.1.2.1" => :cur_tx_mod_rate, + "#{@af_base}.2.1.18.1" => :remote_tx_mod_rate, + "#{@af_base}.2.1.5.1" => :rx_capacity, + "#{@af_base}.2.1.6.1" => :tx_capacity, + "#{@af_base}.2.1.17.1" => :remote_tx_power, + "#{@af_base}.2.1.19.1" => :remote_rx_power_0, + "#{@af_base}.2.1.22.1" => :remote_rx_power_1, + "#{@af_base}.2.1.26.1" => :link_state, + "#{@af_base}.2.1.44.1" => :link_uptime, + "#{@af_base}.2.1.8.1" => :radio_temp_0_c, + "#{@af_base}.2.1.10.1" => :radio_temp_1_c, + "#{@af_base}.1.1.5.1" => :tx_freq_mhz, + "#{@af_base}.1.1.6.1" => :rx_freq_mhz, + "#{@af_base}.2.1.4.1" => :link_distance_m + } + + # AF60-LR static OIDs + @af60_base "1.3.6.1.4.1.41112.1.11.1" + @af60_static_oids %{ + "#{@af60_base}.2.5.1" => :radio_temp_0_c, + "#{@af60_base}.2.6.1" => :link_state, + "#{@af60_base}.1.2.1" => :tx_freq_mhz + } + + # AF60 station table column -> field name + unit conversion + @af60_station_cols %{ + 3 => {:rx_power_0, :raw}, + 4 => {:tx_power, :raw}, + 5 => {:cur_tx_mod_rate, :raw}, + 6 => {:remote_tx_mod_rate, :raw}, + 7 => {:tx_capacity, :kbps_to_mbps}, + 8 => {:rx_capacity, :kbps_to_mbps}, + 15 => {:link_distance_m, :raw}, + 17 => {:link_uptime, :centiseconds_to_seconds}, + 18 => {:remote_rx_power_0, :raw}, + 19 => {:remote_tx_power, :raw} + } + + @snmp_timeout 5_000 + + def poll(host, community, "af11x"), do: poll_af11x(host, community) + def poll(host, community, "af60"), do: poll_af60(host, community) + + def poll_af11x(host, community) do + oids = Map.keys(@af11x_oids) + args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | oids] + + case run_cmd("snmpget", args) do + {output, 0} -> + result = parse_snmpget_output(output, :af11x) + if map_size(result) > 0, do: {:ok, result}, else: {:error, :empty_response} + + {output, _exit_code} -> + Logger.warning("snmpget failed for #{host}: #{String.trim(output)}") + {:error, :snmp_failed} + end + end + + def poll_af60(host, community) do + static_oids = Map.keys(@af60_static_oids) + static_args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | static_oids] + + station_oids = Enum.map(Map.keys(@af60_station_cols), &"#{@af60_base}.3.1.#{&1}") + station_args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | station_oids] + + with {static_out, 0} <- run_cmd("snmpget", static_args), + {station_out, 0} <- run_cmd("snmpgetnext", station_args) do + result = parse_af60_output(static_out, station_out) + if map_size(result) > 0, do: {:ok, result}, else: {:error, :empty_response} + else + {output, _} -> + Logger.warning("snmp failed for AF60 #{host}: #{String.trim(output)}") + {:error, :snmp_failed} + end + end + + def parse_snmpget_output(output, :af11x) do + output + |> String.split("\n", trim: true) + |> Enum.reduce(%{}, fn line, acc -> + case parse_snmp_line(line) do + {oid, value} -> + case find_af11x_field(oid) do + nil -> acc + field -> Map.put(acc, field, value) + end + + nil -> + acc + end + end) + end + + def parse_af60_output(static_output, station_output) do + static = + static_output + |> String.split("\n", trim: true) + |> Enum.reduce(%{}, fn line, acc -> + case parse_snmp_line(line) do + {oid, value} -> + case find_af60_static_field(oid) do + nil -> acc + field -> Map.put(acc, field, value) + end + + nil -> + acc + end + end) + + station = + station_output + |> String.split("\n", trim: true) + |> Enum.reduce(%{}, fn line, acc -> + case parse_snmp_line(line) do + {oid, value} -> + case find_af60_station_field(oid) do + nil -> acc + {field, conversion} -> Map.put(acc, field, convert_value(value, conversion)) + end + + nil -> + acc + end + end) + + Map.merge(static, station) + end + + # --- Private --- + + defp run_cmd(cmd, args) do + System.cmd(cmd, args, stderr_to_stdout: true, timeout: @snmp_timeout) + rescue + ErlangError -> {"command not found: #{cmd}", 127} + end + + defp parse_snmp_line(line) do + # Format: "iso.3.6.1.4... = INTEGER: -55" or "...= No Such Instance..." + case String.split(line, " = ", parts: 2) do + [oid_part, value_part] -> + if String.contains?(value_part, "No Such") do + nil + else + oid = oid_part |> String.trim() |> normalize_oid() + value = parse_snmp_value(value_part) + if value, do: {oid, value}, else: nil + end + + _ -> + nil + end + end + + defp normalize_oid(oid) do + oid + |> String.replace(~r/^iso\./, "1.") + |> String.replace(~r/^\./, "") + end + + defp parse_snmp_value(value_str) do + # "INTEGER: -55" or "Gauge32: 100" or "STRING: \"foo\"" + case Regex.run(~r/:\s*(.+)$/, String.trim(value_str)) do + [_, raw] -> + raw = String.trim(raw, "\"") + + case Integer.parse(raw) do + {val, _} -> val + :error -> nil + end + + nil -> + nil + end + end + + defp find_af11x_field(oid) do + Enum.find_value(@af11x_oids, fn {known_oid, field} -> + if oid == known_oid or String.starts_with?(oid, known_oid), do: field + end) + end + + defp find_af60_static_field(oid) do + Enum.find_value(@af60_static_oids, fn {known_oid, field} -> + if oid == known_oid or String.starts_with?(oid, known_oid), do: field + end) + end + + defp find_af60_station_field(oid) do + # Station table OIDs: {base}.3.1.{col}.{mac_bytes} + Enum.find_value(@af60_station_cols, fn {col, field_and_conv} -> + prefix = "#{@af60_base}.3.1.#{col}." + if String.starts_with?(oid, prefix), do: field_and_conv + end) + end + + defp convert_value(value, :raw), do: value + defp convert_value(value, :kbps_to_mbps), do: div(value, 1000) + defp convert_value(value, :centiseconds_to_seconds), do: div(value, 100) +end +``` + +**Step 4: Run tests** + +Run: `mix test test/microwaveprop/commercial/snmp_client_test.exs` +Expected: All tests pass + +**Step 5: Commit** + +``` +feat: add SNMP client for AF11X and AF60-LR radio polling +``` + +--- + +## Task 6: Poll Worker + +**Files:** +- Create: `lib/microwaveprop/commercial/poll_worker.ex` +- Create: `test/microwaveprop/commercial/poll_worker_test.exs` +- Modify: `config/config.exs` (add queue + cron) +- Modify: `config/test.exs` (add snmp test config) + +**Step 1: Write the failing test** + +```elixir +# test/microwaveprop/commercial/poll_worker_test.exs +defmodule Microwaveprop.Commercial.PollWorkerTest do + use Microwaveprop.DataCase, async: false + + alias Microwaveprop.Commercial + alias Microwaveprop.Commercial.PollWorker + alias Microwaveprop.Commercial.Sample + + @link_attrs %{ + label: "test-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x", + weather_station: "KTKI", + enabled: true + } + + describe "perform/1" do + test "creates samples for enabled links when SNMP succeeds" do + {:ok, link} = Commercial.create_link(@link_attrs) + + # Stub the SNMP client to return success + Req.Test.stub(Microwaveprop.Commercial.SnmpClient, fn _conn -> + Req.Test.json(%{}, %{}) + end) + + # Use a test-friendly approach: override poll function + snmp_result = %{ + rx_power_0: -55, + tx_power: 20, + link_state: 1, + link_uptime: 86400 + } + + PollWorker.poll_and_record([link], fn _host, _community, _type -> {:ok, snmp_result} end) + + samples = Repo.all(Sample) + assert length(samples) == 1 + assert hd(samples).rx_power_0 == -55.0 + assert hd(samples).link_id == link.id + end + + test "skips links when SNMP fails" do + {:ok, _link} = Commercial.create_link(@link_attrs) + + PollWorker.poll_and_record( + Commercial.enabled_links(), + fn _host, _community, _type -> {:error, :snmp_failed} end + ) + + assert Repo.all(Sample) == [] + end + + test "continues polling remaining links after one failure" do + {:ok, _link1} = Commercial.create_link(@link_attrs) + {:ok, _link2} = Commercial.create_link(%{@link_attrs | label: "link-2", host: "10.0.0.2"}) + + call_count = :counters.new(1, [:atomics]) + + poll_fn = fn _host, _community, _type -> + n = :counters.get(call_count, 1) + :counters.add(call_count, 1, 1) + + if n == 0 do + {:error, :snmp_failed} + else + {:ok, %{rx_power_0: -60, link_state: 1}} + end + end + + PollWorker.poll_and_record(Commercial.enabled_links(), poll_fn) + + assert length(Repo.all(Sample)) == 1 + end + end +end +``` + +**Step 2: Run test to verify it fails** + +Run: `mix test test/microwaveprop/commercial/poll_worker_test.exs` +Expected: Compilation error — `Microwaveprop.Commercial.PollWorker` not found + +**Step 3: Write the poll worker** + +```elixir +# lib/microwaveprop/commercial/poll_worker.ex +defmodule Microwaveprop.Commercial.PollWorker do + @moduledoc false + use Oban.Worker, queue: :commercial, max_attempts: 1 + + require Logger + + alias Microwaveprop.Commercial + alias Microwaveprop.Commercial.SnmpClient + alias Microwaveprop.Weather + alias Microwaveprop.Weather.IemClient + + @impl Oban.Worker + def perform(%Oban.Job{}) do + links = Commercial.enabled_links() + + poll_and_record(links, &SnmpClient.poll/3) + fetch_weather(links) + + :ok + end + + def poll_and_record(links, poll_fn) do + now = DateTime.utc_now() + + Enum.each(links, fn link -> + case poll_fn.(link.host, link.community, link.radio_type) do + {:ok, data} -> + attrs = + data + |> Map.put(:link_id, link.id) + |> Map.put(:sampled_at, now) + + case Commercial.create_sample(attrs) do + {:ok, _sample} -> + Logger.info("Recorded sample for #{link.label}") + + {:error, changeset} -> + Logger.warning("Failed to save sample for #{link.label}: #{inspect(changeset.errors)}") + end + + {:error, reason} -> + Logger.warning("SNMP poll failed for #{link.label} (#{link.host}): #{inspect(reason)}") + end + end) + end + + defp fetch_weather(links) do + stations = + links + |> Enum.map(& &1.weather_station) + |> Enum.reject(&is_nil/1) + |> Enum.uniq() + + now = DateTime.utc_now() + start_dt = DateTime.add(now, -3600, :second) + + Enum.each(stations, fn station_code -> + {:ok, station} = + Weather.find_or_create_station(%{ + station_code: station_code, + station_type: "asos", + name: station_code, + lat: 0.0, + lon: 0.0 + }) + + case IemClient.fetch_asos(station_code, start_dt, now) do + {:ok, rows} -> + Enum.each(rows, fn row -> + if row.observed_at, do: Weather.upsert_surface_observation(station, row) + end) + + Logger.info("Fetched #{length(rows)} ASOS observations for #{station_code}") + + {:error, reason} -> + Logger.warning("ASOS fetch failed for #{station_code}: #{inspect(reason)}") + end + end) + end +end +``` + +**Step 4: Update config/config.exs — add the commercial queue and cron entry** + +In `config/config.exs`, update the Oban config: + +Change: +```elixir + queues: [solar: 1, weather: 3, enqueue: 1, hrrr: 20, terrain: 4], +``` +To: +```elixir + queues: [solar: 1, weather: 3, enqueue: 1, hrrr: 20, terrain: 4, commercial: 2], +``` + +And add to the crontab: +```elixir + {"*/5 * * * *", Microwaveprop.Commercial.PollWorker} +``` + +**Step 5: Run tests** + +Run: `mix test test/microwaveprop/commercial/poll_worker_test.exs` +Expected: All tests pass + +**Step 6: Commit** + +``` +feat: add commercial poll worker with 5-minute cron schedule +``` + +--- + +## Task 7: Dockerfile — Add net-snmp + +**Files:** +- Modify: `Dockerfile` + +**Step 1: Add snmp-utils to the runner image** + +In the `FROM ${RUNNER_IMAGE} AS final` section, add `snmp` to the `apt-get install` line: + +Change: +```dockerfile +RUN apt-get update \ + && apt-get install -y --no-install-recommends libstdc++6 openssl libncurses6 locales ca-certificates \ + && rm -rf /var/lib/apt/lists/* +``` + +To: +```dockerfile +RUN apt-get update \ + && apt-get install -y --no-install-recommends libstdc++6 openssl libncurses6 locales ca-certificates snmp \ + && rm -rf /var/lib/apt/lists/* +``` + +The `snmp` package provides `snmpget` and `snmpgetnext` on Debian. + +**Step 2: Commit** + +``` +feat: add net-snmp to Docker image for commercial link polling +``` + +--- + +## Task 8: Integration — Run precommit + +**Step 1: Format** + +Run: `mix format` +Note: Styler will rewrite code. Let it. + +**Step 2: Run precommit** + +Run: `mix precommit` +Expected: Compiles with no warnings, format clean, all tests pass + +**Step 3: Fix any issues** + +If Styler rewrites anything (e.g., `with` to `case`, pipe adjustments), accept the changes. If there are warnings-as-errors, fix them. + +**Step 4: Final commit if needed** + +``` +chore: fix formatting and warnings from precommit +``` diff --git a/lib/microwaveprop/commercial.ex b/lib/microwaveprop/commercial.ex new file mode 100644 index 00000000..e788d252 --- /dev/null +++ b/lib/microwaveprop/commercial.ex @@ -0,0 +1,35 @@ +defmodule Microwaveprop.Commercial do + @moduledoc false + + import Ecto.Query + + alias Microwaveprop.Commercial.Link + alias Microwaveprop.Commercial.Sample + alias Microwaveprop.Repo + + def enabled_links do + Link + |> where([l], l.enabled == true) + |> Repo.all() + end + + def create_link(attrs) do + %Link{} + |> Link.changeset(attrs) + |> Repo.insert() + end + + def create_sample(attrs) do + %Sample{} + |> Sample.changeset(attrs) + |> Repo.insert() + end + + def weather_stations do + Link + |> where([l], l.enabled == true and not is_nil(l.weather_station)) + |> select([l], l.weather_station) + |> distinct(true) + |> Repo.all() + end +end diff --git a/lib/microwaveprop/commercial/link.ex b/lib/microwaveprop/commercial/link.ex new file mode 100644 index 00000000..47365aef --- /dev/null +++ b/lib/microwaveprop/commercial/link.ex @@ -0,0 +1,33 @@ +defmodule Microwaveprop.Commercial.Link do + @moduledoc false + use Ecto.Schema + + import Ecto.Changeset + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + + schema "commercial_links" do + field :label, :string + field :host, :string + field :community, :string + field :radio_type, :string, default: "af11x" + field :endpoint_a, :map + field :endpoint_b, :map + field :weather_station, :string + field :enabled, :boolean, default: true + + timestamps(type: :utc_datetime) + end + + @required_fields ~w(label host community radio_type)a + @optional_fields ~w(endpoint_a endpoint_b weather_station enabled)a + + def changeset(link, attrs) do + link + |> cast(attrs, @required_fields ++ @optional_fields) + |> validate_required(@required_fields) + |> validate_inclusion(:radio_type, ~w(af11x af60)) + |> unique_constraint(:label) + end +end diff --git a/lib/microwaveprop/commercial/poll_worker.ex b/lib/microwaveprop/commercial/poll_worker.ex new file mode 100644 index 00000000..ad7f0b0f --- /dev/null +++ b/lib/microwaveprop/commercial/poll_worker.ex @@ -0,0 +1,80 @@ +defmodule Microwaveprop.Commercial.PollWorker do + @moduledoc false + use Oban.Worker, queue: :commercial, max_attempts: 1 + + alias Microwaveprop.Commercial + alias Microwaveprop.Commercial.SnmpClient + alias Microwaveprop.Weather + alias Microwaveprop.Weather.IemClient + + require Logger + + @impl Oban.Worker + def perform(%Oban.Job{}) do + links = Commercial.enabled_links() + + poll_and_record(links, &SnmpClient.poll/3) + fetch_weather(links) + + :ok + end + + def poll_and_record(links, poll_fn) do + now = DateTime.utc_now() + + Enum.each(links, fn link -> + case poll_fn.(link.host, link.community, link.radio_type) do + {:ok, data} -> + attrs = + data + |> Map.put(:link_id, link.id) + |> Map.put(:sampled_at, now) + + case Commercial.create_sample(attrs) do + {:ok, _sample} -> + Logger.info("Recorded sample for #{link.label}") + + {:error, changeset} -> + Logger.warning("Failed to save sample for #{link.label}: #{inspect(changeset.errors)}") + end + + {:error, reason} -> + Logger.warning("SNMP poll failed for #{link.label} (#{link.host}): #{inspect(reason)}") + end + end) + end + + defp fetch_weather(links) do + stations = + links + |> Enum.map(& &1.weather_station) + |> Enum.reject(&is_nil/1) + |> Enum.uniq() + + now = DateTime.utc_now() + start_dt = DateTime.add(now, -3600, :second) + + Enum.each(stations, fn station_code -> + {:ok, station} = + Weather.find_or_create_station(%{ + station_code: station_code, + station_type: "asos", + name: station_code, + lat: 0.0, + lon: 0.0 + }) + + case IemClient.fetch_asos(station_code, start_dt, now) do + {:ok, rows} -> + Enum.each(rows, fn row -> + if row.observed_at, do: Weather.upsert_surface_observation(station, row) + end) + + Logger.info("Fetched #{length(rows)} ASOS observations for #{station_code}") + + {:error, reason} -> + Logger.warning("ASOS fetch failed for #{station_code}: #{inspect(reason)}") + end + end) + end +end diff --git a/lib/microwaveprop/commercial/sample.ex b/lib/microwaveprop/commercial/sample.ex new file mode 100644 index 00000000..ac9d632a --- /dev/null +++ b/lib/microwaveprop/commercial/sample.ex @@ -0,0 +1,54 @@ +defmodule Microwaveprop.Commercial.Sample do + @moduledoc false + use Ecto.Schema + + import Ecto.Changeset + + @primary_key {:id, :binary_id, autogenerate: true} + @foreign_key_type :binary_id + + schema "commercial_samples" do + belongs_to :link, Microwaveprop.Commercial.Link + field :sampled_at, :utc_datetime_usec + + # Local radio + field :rx_power_0, :float + field :rx_power_1, :float + field :tx_power, :float + field :tx_freq_mhz, :integer + field :rx_freq_mhz, :integer + field :radio_temp_0_c, :float + field :radio_temp_1_c, :float + + # Modulation & capacity + field :cur_tx_mod_rate, :integer + field :remote_tx_mod_rate, :integer + field :rx_capacity, :integer + field :tx_capacity, :integer + + # Remote radio + field :remote_tx_power, :float + field :remote_rx_power_0, :float + field :remote_rx_power_1, :float + + # Link status + field :link_state, :integer + field :link_uptime, :integer + field :link_distance_m, :integer + + timestamps(type: :utc_datetime) + end + + @required_fields ~w(link_id sampled_at)a + @optional_fields ~w(rx_power_0 rx_power_1 tx_power tx_freq_mhz rx_freq_mhz + radio_temp_0_c radio_temp_1_c cur_tx_mod_rate remote_tx_mod_rate + rx_capacity tx_capacity remote_tx_power remote_rx_power_0 remote_rx_power_1 + link_state link_uptime link_distance_m)a + + def changeset(sample, attrs) do + sample + |> cast(attrs, @required_fields ++ @optional_fields) + |> validate_required(@required_fields) + |> foreign_key_constraint(:link_id) + end +end diff --git a/lib/microwaveprop/commercial/snmp_client.ex b/lib/microwaveprop/commercial/snmp_client.ex new file mode 100644 index 00000000..1a97cc20 --- /dev/null +++ b/lib/microwaveprop/commercial/snmp_client.ex @@ -0,0 +1,198 @@ +defmodule Microwaveprop.Commercial.SnmpClient do + @moduledoc false + + require Logger + + # UBNT-AirFIBER-MIB OIDs (base 1.3.6.1.4.1.41112.1.3) + @af_base "1.3.6.1.4.1.41112.1.3" + @af11x_oids %{ + "#{@af_base}.2.1.11.1" => :rx_power_0, + "#{@af_base}.2.1.14.1" => :rx_power_1, + "#{@af_base}.1.1.9.1" => :tx_power, + "#{@af_base}.2.1.2.1" => :cur_tx_mod_rate, + "#{@af_base}.2.1.18.1" => :remote_tx_mod_rate, + "#{@af_base}.2.1.5.1" => :rx_capacity, + "#{@af_base}.2.1.6.1" => :tx_capacity, + "#{@af_base}.2.1.17.1" => :remote_tx_power, + "#{@af_base}.2.1.19.1" => :remote_rx_power_0, + "#{@af_base}.2.1.22.1" => :remote_rx_power_1, + "#{@af_base}.2.1.26.1" => :link_state, + "#{@af_base}.2.1.44.1" => :link_uptime, + "#{@af_base}.2.1.8.1" => :radio_temp_0_c, + "#{@af_base}.2.1.10.1" => :radio_temp_1_c, + "#{@af_base}.1.1.5.1" => :tx_freq_mhz, + "#{@af_base}.1.1.6.1" => :rx_freq_mhz, + "#{@af_base}.2.1.4.1" => :link_distance_m + } + + # AF60-LR static OIDs + @af60_base "1.3.6.1.4.1.41112.1.11.1" + @af60_static_oids %{ + "#{@af60_base}.2.5.1" => :radio_temp_0_c, + "#{@af60_base}.2.6.1" => :link_state, + "#{@af60_base}.1.2.1" => :tx_freq_mhz + } + + # AF60 station table column -> field name + unit conversion + @af60_station_cols %{ + 3 => {:rx_power_0, :raw}, + 4 => {:tx_power, :raw}, + 5 => {:cur_tx_mod_rate, :raw}, + 6 => {:remote_tx_mod_rate, :raw}, + 7 => {:tx_capacity, :kbps_to_mbps}, + 8 => {:rx_capacity, :kbps_to_mbps}, + 15 => {:link_distance_m, :raw}, + 17 => {:link_uptime, :centiseconds_to_seconds}, + 18 => {:remote_rx_power_0, :raw}, + 19 => {:remote_tx_power, :raw} + } + + @snmp_timeout 5_000 + + def poll(host, community, "af11x"), do: poll_af11x(host, community) + def poll(host, community, "af60"), do: poll_af60(host, community) + + def poll_af11x(host, community) do + oids = Map.keys(@af11x_oids) + args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | oids] + + case run_cmd("snmpget", args) do + {output, 0} -> + result = parse_snmpget_output(output, :af11x) + if map_size(result) > 0, do: {:ok, result}, else: {:error, :empty_response} + + {output, _exit_code} -> + Logger.warning("snmpget failed for #{host}: #{String.trim(output)}") + {:error, :snmp_failed} + end + end + + def poll_af60(host, community) do + static_oids = Map.keys(@af60_static_oids) + static_args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | static_oids] + + station_oids = Enum.map(Map.keys(@af60_station_cols), &"#{@af60_base}.3.1.#{&1}") + station_args = ["-v2c", "-c", community, "-t", "5", "-r", "1", host | station_oids] + + with {static_out, 0} <- run_cmd("snmpget", static_args), + {station_out, 0} <- run_cmd("snmpgetnext", station_args) do + result = parse_af60_output(static_out, station_out) + if map_size(result) > 0, do: {:ok, result}, else: {:error, :empty_response} + else + {output, _} -> + Logger.warning("snmp failed for AF60 #{host}: #{String.trim(output)}") + {:error, :snmp_failed} + end + end + + def parse_snmpget_output(output, :af11x) do + output + |> parse_lines() + |> Enum.reduce(%{}, fn {oid, value}, acc -> + case find_af11x_field(oid) do + nil -> acc + field -> Map.put(acc, field, value) + end + end) + end + + def parse_af60_output(static_output, station_output) do + static = + static_output + |> parse_lines() + |> Enum.reduce(%{}, fn {oid, value}, acc -> + case find_af60_static_field(oid) do + nil -> acc + field -> Map.put(acc, field, value) + end + end) + + station = + station_output + |> parse_lines() + |> Enum.reduce(%{}, fn {oid, value}, acc -> + case find_af60_station_field(oid) do + nil -> acc + {field, conversion} -> Map.put(acc, field, convert_value(value, conversion)) + end + end) + + Map.merge(static, station) + end + + # --- Private --- + + defp parse_lines(output) do + output + |> String.split("\n", trim: true) + |> Enum.flat_map(fn line -> + case parse_snmp_line(line) do + {_oid, _value} = pair -> [pair] + nil -> [] + end + end) + end + + defp run_cmd(cmd, args) do + System.cmd(cmd, args, stderr_to_stdout: true, timeout: @snmp_timeout) + rescue + ErlangError -> {"command not found: #{cmd}", 127} + end + + defp parse_snmp_line(line) do + line = String.trim(line) + + with [oid_part, value_part] <- String.split(line, " = ", parts: 2), + false <- String.contains?(value_part, "No Such"), + oid = oid_part |> String.trim() |> normalize_oid(), + value when not is_nil(value) <- parse_snmp_value(value_part) do + {oid, value} + else + _ -> nil + end + end + + defp normalize_oid(oid) do + oid + |> String.replace(~r/^iso\./, "1.") + |> String.replace(~r/^\./, "") + end + + defp parse_snmp_value(value_str) do + case Regex.run(~r/:\s*(.+)$/, String.trim(value_str)) do + [_, raw] -> + raw = String.trim(raw, "\"") + + case Integer.parse(raw) do + {val, _} -> val + :error -> nil + end + + nil -> + nil + end + end + + defp find_af11x_field(oid) do + Enum.find_value(@af11x_oids, fn {known_oid, field} -> + if oid == known_oid, do: field + end) + end + + defp find_af60_static_field(oid) do + Enum.find_value(@af60_static_oids, fn {known_oid, field} -> + if oid == known_oid, do: field + end) + end + + defp find_af60_station_field(oid) do + Enum.find_value(@af60_station_cols, fn {col, field_and_conv} -> + prefix = "#{@af60_base}.3.1.#{col}." + if String.starts_with?(oid, prefix), do: field_and_conv + end) + end + + defp convert_value(value, :raw), do: value + defp convert_value(value, :kbps_to_mbps), do: div(value, 1000) + defp convert_value(value, :centiseconds_to_seconds), do: div(value, 100) +end diff --git a/priv/repo/migrations/20260330174211_create_commercial_links.exs b/priv/repo/migrations/20260330174211_create_commercial_links.exs new file mode 100644 index 00000000..3f65f6c8 --- /dev/null +++ b/priv/repo/migrations/20260330174211_create_commercial_links.exs @@ -0,0 +1,22 @@ +defmodule Microwaveprop.Repo.Migrations.CreateCommercialLinks do + use Ecto.Migration + + def change do + create table(:commercial_links, primary_key: false) do + add :id, :binary_id, primary_key: true + add :label, :string, null: false + add :host, :string, null: false + add :community, :string, null: false + add :radio_type, :string, null: false, default: "af11x" + add :endpoint_a, :map + add :endpoint_b, :map + add :weather_station, :string + add :enabled, :boolean, null: false, default: true + + timestamps(type: :utc_datetime) + end + + create unique_index(:commercial_links, [:label]) + create index(:commercial_links, [:enabled]) + end +end diff --git a/priv/repo/migrations/20260330174511_seed_commercial_links.exs b/priv/repo/migrations/20260330174511_seed_commercial_links.exs new file mode 100644 index 00000000..84891500 --- /dev/null +++ b/priv/repo/migrations/20260330174511_seed_commercial_links.exs @@ -0,0 +1,85 @@ +defmodule Microwaveprop.Repo.Migrations.SeedCommercialLinks do + use Ecto.Migration + + def up do + now = DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_naive() + + links = [ + %{ + label: "verona-to-climax", + host: "10.250.1.26", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.246171,"lon":-96.426516,"alt_m":205.4}|, + endpoint_b: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + weather_station: "KTKI" + }, + %{ + label: "new-hope-to-core", + host: "10.250.1.58", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.207279,"lon":-96.537762,"alt_m":216.4}|, + endpoint_b: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + weather_station: "KTKI" + }, + %{ + label: "core-new-hope", + host: "10.250.1.61", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + endpoint_b: ~s|{"lat":33.207279,"lon":-96.537762,"alt_m":216.4}|, + weather_station: "KTKI" + }, + %{ + label: "982_380_60LR", + host: "10.250.1.34", + community: "kdyyJrT0Mm", + radio_type: "af60", + endpoint_a: ~s|{"lat":33.148921,"lon":-96.4958,"alt_m":186.5}|, + endpoint_b: ~s|{"lat":33.174135,"lon":-96.491935,"alt_m":200.4}|, + weather_station: "KTKI" + }, + %{ + label: "380_982_60LR", + host: "10.250.1.37", + community: "kdyyJrT0Mm", + radio_type: "af60", + endpoint_a: ~s|{"lat":33.174135,"lon":-96.491935,"alt_m":200.4}|, + endpoint_b: ~s|{"lat":33.148921,"lon":-96.4958,"alt_m":186.5}|, + weather_station: "KTKI" + }, + %{ + label: "core-to-climax", + host: "10.250.1.93", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + endpoint_b: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + weather_station: "KTKI" + }, + %{ + label: "climax-to-core", + host: "10.250.1.90", + community: "kdyyJrT0Mm", + radio_type: "af11x", + endpoint_a: ~s|{"lat":33.187294,"lon":-96.448128,"alt_m":191.0}|, + endpoint_b: ~s|{"lat":33.174149,"lon":-96.491943,"alt_m":198.9}|, + weather_station: "KTKI" + } + ] + + for link <- links do + execute(""" + INSERT INTO commercial_links (id, label, host, community, radio_type, endpoint_a, endpoint_b, weather_station, enabled, inserted_at, updated_at) + VALUES (gen_random_uuid(), '#{link.label}', '#{link.host}', '#{link.community}', '#{link.radio_type}', '#{link.endpoint_a}'::jsonb, '#{link.endpoint_b}'::jsonb, '#{link.weather_station}', true, '#{now}', '#{now}') + ON CONFLICT (label) DO NOTHING + """) + end + end + + def down do + execute("DELETE FROM commercial_links") + end +end diff --git a/priv/repo/migrations/20260330174723_create_commercial_samples.exs b/priv/repo/migrations/20260330174723_create_commercial_samples.exs new file mode 100644 index 00000000..de40cca9 --- /dev/null +++ b/priv/repo/migrations/20260330174723_create_commercial_samples.exs @@ -0,0 +1,44 @@ +defmodule Microwaveprop.Repo.Migrations.CreateCommercialSamples do + use Ecto.Migration + + def change do + create table(:commercial_samples, primary_key: false) do + add :id, :binary_id, primary_key: true + + add :link_id, references(:commercial_links, type: :binary_id, on_delete: :delete_all), + null: false + + add :sampled_at, :utc_datetime_usec, null: false + + # Local radio + add :rx_power_0, :float + add :rx_power_1, :float + add :tx_power, :float + add :tx_freq_mhz, :integer + add :rx_freq_mhz, :integer + add :radio_temp_0_c, :float + add :radio_temp_1_c, :float + + # Modulation & capacity + add :cur_tx_mod_rate, :integer + add :remote_tx_mod_rate, :integer + add :rx_capacity, :integer + add :tx_capacity, :integer + + # Remote radio + add :remote_tx_power, :float + add :remote_rx_power_0, :float + add :remote_rx_power_1, :float + + # Link status + add :link_state, :integer + add :link_uptime, :integer + add :link_distance_m, :integer + + timestamps(type: :utc_datetime) + end + + create index(:commercial_samples, [:link_id, :sampled_at]) + create index(:commercial_samples, [:sampled_at]) + end +end diff --git a/test/microwaveprop/commercial/link_test.exs b/test/microwaveprop/commercial/link_test.exs new file mode 100644 index 00000000..5e56a86e --- /dev/null +++ b/test/microwaveprop/commercial/link_test.exs @@ -0,0 +1,58 @@ +defmodule Microwaveprop.Commercial.LinkTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial.Link + + @valid_attrs %{ + label: "test-link-abc", + host: "10.250.1.26", + community: "public", + radio_type: "af11x", + endpoint_a: %{lat: 33.246171, lon: -96.426516, alt_m: 205.4}, + endpoint_b: %{lat: 33.187294, lon: -96.448128, alt_m: 191.0}, + weather_station: "KTKI", + enabled: true + } + + describe "changeset/2" do + test "valid attributes" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert changeset.valid? + end + + test "requires label, host, community" do + changeset = Link.changeset(%Link{}, %{}) + + assert %{ + label: ["can't be blank"], + host: ["can't be blank"], + community: ["can't be blank"] + } = errors_on(changeset) + end + + test "validates radio_type inclusion" do + changeset = Link.changeset(%Link{}, %{@valid_attrs | radio_type: "unknown"}) + assert %{radio_type: ["is invalid"]} = errors_on(changeset) + end + + test "defaults enabled to true" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert Ecto.Changeset.get_field(changeset, :enabled) == true + end + + test "persists to the database" do + changeset = Link.changeset(%Link{}, @valid_attrs) + assert {:ok, link} = Repo.insert(changeset) + assert link.label == "test-link-abc" + assert link.host == "10.250.1.26" + assert link.radio_type == "af11x" + assert link.endpoint_a[:lat] == 33.246171 + end + + test "enforces unique label" do + assert {:ok, _} = Repo.insert(Link.changeset(%Link{}, @valid_attrs)) + assert {:error, changeset} = Repo.insert(Link.changeset(%Link{}, @valid_attrs)) + assert %{label: ["has already been taken"]} = errors_on(changeset) + end + end +end diff --git a/test/microwaveprop/commercial/poll_worker_test.exs b/test/microwaveprop/commercial/poll_worker_test.exs new file mode 100644 index 00000000..7a5d0505 --- /dev/null +++ b/test/microwaveprop/commercial/poll_worker_test.exs @@ -0,0 +1,69 @@ +defmodule Microwaveprop.Commercial.PollWorkerTest do + use Microwaveprop.DataCase, async: false + + alias Microwaveprop.Commercial + alias Microwaveprop.Commercial.PollWorker + alias Microwaveprop.Commercial.Sample + + @link_attrs %{ + label: "test-poll-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x", + weather_station: "KTKI", + enabled: true + } + + describe "poll_and_record/2" do + test "creates samples for enabled links when SNMP succeeds" do + {:ok, link} = Commercial.create_link(@link_attrs) + + snmp_result = %{ + rx_power_0: -55, + tx_power: 20, + link_state: 1, + link_uptime: 86_400 + } + + PollWorker.poll_and_record([link], fn _host, _community, _type -> {:ok, snmp_result} end) + + samples = Repo.all(Sample) + assert length(samples) == 1 + assert hd(samples).rx_power_0 == -55.0 + assert hd(samples).link_id == link.id + end + + test "skips links when SNMP fails" do + {:ok, link} = Commercial.create_link(@link_attrs) + + PollWorker.poll_and_record( + [link], + fn _host, _community, _type -> {:error, :snmp_failed} end + ) + + assert Repo.all(Sample) == [] + end + + test "continues polling remaining links after one failure" do + {:ok, link1} = Commercial.create_link(@link_attrs) + {:ok, link2} = Commercial.create_link(%{@link_attrs | label: "test-poll-link-2", host: "10.0.0.2"}) + + call_count = :counters.new(1, [:atomics]) + + poll_fn = fn _host, _community, _type -> + n = :counters.get(call_count, 1) + :counters.add(call_count, 1, 1) + + if n == 0 do + {:error, :snmp_failed} + else + {:ok, %{rx_power_0: -60, link_state: 1}} + end + end + + PollWorker.poll_and_record([link1, link2], poll_fn) + + assert length(Repo.all(Sample)) == 1 + end + end +end diff --git a/test/microwaveprop/commercial/sample_test.exs b/test/microwaveprop/commercial/sample_test.exs new file mode 100644 index 00000000..1508f7b7 --- /dev/null +++ b/test/microwaveprop/commercial/sample_test.exs @@ -0,0 +1,58 @@ +defmodule Microwaveprop.Commercial.SampleTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial.Link + alias Microwaveprop.Commercial.Sample + + defp create_link do + %Link{} + |> Link.changeset(%{ + label: "test-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x" + }) + |> Repo.insert!() + end + + @sample_attrs %{ + sampled_at: ~U[2026-03-30 12:00:00.000000Z], + rx_power_0: -55.0, + tx_power: 20.0, + link_state: 1, + link_uptime: 86_400, + cur_tx_mod_rate: 6, + rx_capacity: 450, + tx_capacity: 450 + } + + describe "changeset/2" do + test "valid attributes with link_id" do + link = create_link() + attrs = Map.put(@sample_attrs, :link_id, link.id) + changeset = Sample.changeset(%Sample{}, attrs) + assert changeset.valid? + end + + test "requires link_id and sampled_at" do + changeset = Sample.changeset(%Sample{}, %{}) + assert %{link_id: ["can't be blank"], sampled_at: ["can't be blank"]} = errors_on(changeset) + end + + test "persists to the database" do + link = create_link() + attrs = Map.put(@sample_attrs, :link_id, link.id) + + assert {:ok, sample} = Repo.insert(Sample.changeset(%Sample{}, attrs)) + assert sample.rx_power_0 == -55.0 + assert sample.link_id == link.id + end + + test "all signal fields are optional" do + link = create_link() + attrs = %{link_id: link.id, sampled_at: ~U[2026-03-30 12:00:00.000000Z]} + changeset = Sample.changeset(%Sample{}, attrs) + assert changeset.valid? + end + end +end diff --git a/test/microwaveprop/commercial/snmp_client_test.exs b/test/microwaveprop/commercial/snmp_client_test.exs new file mode 100644 index 00000000..8ff0e7c1 --- /dev/null +++ b/test/microwaveprop/commercial/snmp_client_test.exs @@ -0,0 +1,103 @@ +defmodule Microwaveprop.Commercial.SnmpClientTest do + use ExUnit.Case, async: true + + alias Microwaveprop.Commercial.SnmpClient + + @af11x_output """ + iso.3.6.1.4.1.41112.1.3.2.1.11.1 = INTEGER: -55 + iso.3.6.1.4.1.41112.1.3.2.1.14.1 = INTEGER: -56 + iso.3.6.1.4.1.41112.1.3.1.1.9.1 = INTEGER: 20 + iso.3.6.1.4.1.41112.1.3.2.1.2.1 = INTEGER: 6 + iso.3.6.1.4.1.41112.1.3.2.1.18.1 = INTEGER: 6 + iso.3.6.1.4.1.41112.1.3.2.1.5.1 = INTEGER: 450 + iso.3.6.1.4.1.41112.1.3.2.1.6.1 = INTEGER: 450 + iso.3.6.1.4.1.41112.1.3.2.1.17.1 = INTEGER: 20 + iso.3.6.1.4.1.41112.1.3.2.1.19.1 = INTEGER: -54 + iso.3.6.1.4.1.41112.1.3.2.1.22.1 = INTEGER: -57 + iso.3.6.1.4.1.41112.1.3.2.1.26.1 = INTEGER: 1 + iso.3.6.1.4.1.41112.1.3.2.1.44.1 = INTEGER: 86400 + iso.3.6.1.4.1.41112.1.3.2.1.8.1 = INTEGER: 45 + iso.3.6.1.4.1.41112.1.3.2.1.10.1 = INTEGER: 47 + iso.3.6.1.4.1.41112.1.3.1.1.5.1 = INTEGER: 11245 + iso.3.6.1.4.1.41112.1.3.1.1.6.1 = INTEGER: 10720 + iso.3.6.1.4.1.41112.1.3.2.1.4.1 = INTEGER: 7200 + """ + + describe "parse_snmpget_output/2" do + test "parses AF11X snmpget output into signal map" do + result = SnmpClient.parse_snmpget_output(@af11x_output, :af11x) + + assert result.rx_power_0 == -55 + assert result.rx_power_1 == -56 + assert result.tx_power == 20 + assert result.cur_tx_mod_rate == 6 + assert result.remote_tx_mod_rate == 6 + assert result.rx_capacity == 450 + assert result.tx_capacity == 450 + assert result.remote_tx_power == 20 + assert result.remote_rx_power_0 == -54 + assert result.remote_rx_power_1 == -57 + assert result.link_state == 1 + assert result.link_uptime == 86_400 + assert result.radio_temp_0_c == 45 + assert result.radio_temp_1_c == 47 + assert result.tx_freq_mhz == 11_245 + assert result.rx_freq_mhz == 10_720 + assert result.link_distance_m == 7200 + end + + test "returns empty map for empty output" do + assert SnmpClient.parse_snmpget_output("", :af11x) == %{} + end + + test "skips lines with No Such Instance" do + output = """ + iso.3.6.1.4.1.41112.1.3.2.1.11.1 = INTEGER: -55 + iso.3.6.1.4.1.41112.1.3.2.1.14.1 = No Such Instance currently exists at this OID + """ + + result = SnmpClient.parse_snmpget_output(output, :af11x) + assert result.rx_power_0 == -55 + refute Map.has_key?(result, :rx_power_1) + end + end + + @af60_static_output """ + iso.3.6.1.4.1.41112.1.11.1.2.5.1 = INTEGER: 52 + iso.3.6.1.4.1.41112.1.11.1.2.6.1 = INTEGER: 1 + iso.3.6.1.4.1.41112.1.11.1.1.2.1 = INTEGER: 60480 + """ + + @af60_station_output """ + iso.3.6.1.4.1.41112.1.11.1.3.1.3.170.211.109.42.26.137 = INTEGER: -58 + iso.3.6.1.4.1.41112.1.11.1.3.1.4.170.211.109.42.26.137 = INTEGER: 18 + iso.3.6.1.4.1.41112.1.11.1.3.1.5.170.211.109.42.26.137 = INTEGER: 9 + iso.3.6.1.4.1.41112.1.11.1.3.1.6.170.211.109.42.26.137 = INTEGER: 8 + iso.3.6.1.4.1.41112.1.11.1.3.1.7.170.211.109.42.26.137 = INTEGER: 900000 + iso.3.6.1.4.1.41112.1.11.1.3.1.8.170.211.109.42.26.137 = INTEGER: 850000 + iso.3.6.1.4.1.41112.1.11.1.3.1.15.170.211.109.42.26.137 = INTEGER: 2800 + iso.3.6.1.4.1.41112.1.11.1.3.1.17.170.211.109.42.26.137 = INTEGER: 8640000 + iso.3.6.1.4.1.41112.1.11.1.3.1.18.170.211.109.42.26.137 = INTEGER: -57 + iso.3.6.1.4.1.41112.1.11.1.3.1.19.170.211.109.42.26.137 = INTEGER: 19 + """ + + describe "parse_af60_output/2" do + test "parses AF60 static + station output" do + result = SnmpClient.parse_af60_output(@af60_static_output, @af60_station_output) + + assert result.radio_temp_0_c == 52 + assert result.link_state == 1 + assert result.tx_freq_mhz == 60_480 + assert result.rx_power_0 == -58 + assert result.tx_power == 18 + assert result.cur_tx_mod_rate == 9 + assert result.remote_tx_mod_rate == 8 + assert result.tx_capacity == 900 + assert result.rx_capacity == 850 + assert result.link_distance_m == 2800 + assert result.link_uptime == 86_400 + assert result.remote_rx_power_0 == -57 + assert result.remote_tx_power == 19 + end + end +end diff --git a/test/microwaveprop/commercial_test.exs b/test/microwaveprop/commercial_test.exs new file mode 100644 index 00000000..47c35918 --- /dev/null +++ b/test/microwaveprop/commercial_test.exs @@ -0,0 +1,99 @@ +defmodule Microwaveprop.CommercialTest do + use Microwaveprop.DataCase, async: true + + alias Microwaveprop.Commercial + alias Microwaveprop.Commercial.Link + + setup do + Repo.delete_all(Link) + :ok + end + + @link_attrs %{ + label: "test-link", + host: "10.0.0.1", + community: "public", + radio_type: "af11x", + weather_station: "KTKI", + enabled: true + } + + describe "enabled_links/0" do + test "returns only enabled links" do + {:ok, _enabled} = Commercial.create_link(@link_attrs) + + {:ok, _disabled} = + Commercial.create_link(%{@link_attrs | label: "disabled-link", enabled: false}) + + links = Commercial.enabled_links() + assert length(links) == 1 + assert hd(links).label == "test-link" + end + + test "returns empty list when no enabled links" do + assert Commercial.enabled_links() == [] + end + end + + describe "create_link/1" do + test "creates a link with valid attrs" do + assert {:ok, link} = Commercial.create_link(@link_attrs) + assert link.label == "test-link" + assert link.enabled == true + end + + test "returns error for invalid attrs" do + assert {:error, _changeset} = Commercial.create_link(%{}) + end + end + + describe "create_sample/1" do + test "creates a sample for a link" do + {:ok, link} = Commercial.create_link(@link_attrs) + + attrs = %{ + link_id: link.id, + sampled_at: DateTime.utc_now(), + rx_power_0: -55.0, + tx_power: 20.0, + link_state: 1 + } + + assert {:ok, sample} = Commercial.create_sample(attrs) + assert sample.rx_power_0 == -55.0 + assert sample.link_id == link.id + end + end + + describe "weather_stations/0" do + test "returns unique weather stations from enabled links" do + {:ok, _} = Commercial.create_link(@link_attrs) + + {:ok, _} = + Commercial.create_link(%{@link_attrs | label: "link-2", host: "10.0.0.2"}) + + {:ok, _} = + Commercial.create_link(%{ + @link_attrs + | label: "link-3", + host: "10.0.0.3", + weather_station: "KDFW" + }) + + stations = Commercial.weather_stations() + assert Enum.sort(stations) == ["KDFW", "KTKI"] + end + + test "excludes disabled links" do + {:ok, _} = Commercial.create_link(%{@link_attrs | enabled: false}) + + assert Commercial.weather_stations() == [] + end + + test "excludes nil weather stations" do + {:ok, _} = Commercial.create_link(%{@link_attrs | weather_station: nil}) + + assert Commercial.weather_stations() == [] + end + end +end