Graham McIntire
e490f65359
Some checks failed
Build and Push / Build and Push Docker Image (push) Failing after 4m38s
- Add Canadian HRDPS model coverage (to 60°N) - Document Rust pipeline (prop-grid-rs) and its performance characteristics - Add .sgrid/.pgrid binary format details - Note ~6s chain step and 23-band fused pass - Remove Nx/Axon/ExLA ML model references (scaffolding was removed) - Update roadmap: drop model training, add PSKR calibration, propagation alerts - Fix factor count (was 9, is 10) in roadmap item
288 lines
13 KiB
Elixir
288 lines
13 KiB
Elixir
defmodule MicrowavepropWeb.AboutLive do
|
||
@moduledoc "Static `/about` page — project overview, credits, contact info."
|
||
use MicrowavepropWeb, :live_view
|
||
|
||
alias Microwaveprop.Repo
|
||
|
||
@impl true
|
||
def mount(_params, _session, socket) do
|
||
{:ok,
|
||
socket
|
||
|> assign(:page_title, "About")
|
||
|> assign(:stats, fetch_stats())}
|
||
end
|
||
|
||
@impl true
|
||
def render(assigns) do
|
||
~H"""
|
||
<Layouts.app flash={@flash} current_scope={@current_scope} max_width="max-w-4xl">
|
||
<.header>
|
||
About NTMS Propagation Prediction
|
||
<:subtitle>What we're building and why.</:subtitle>
|
||
</.header>
|
||
|
||
<section class="space-y-8 text-sm leading-relaxed">
|
||
<div class="space-y-3">
|
||
<p>
|
||
Line-of-sight is wrong for the microwave bands. Every operator on
|
||
902 MHz and up has made contacts over hills, through trees, and 200 km
|
||
past the horizon — contacts that shouldn't have worked if LOS were the
|
||
whole story. What's actually going on is the lower atmosphere:
|
||
temperature inversions, humidity gradients, ducting layers, and rain
|
||
cells that bend, trap, and absorb signals in ways that change by the
|
||
hour. The window for a good contact is often minutes, not days.
|
||
</p>
|
||
<p>
|
||
Jim KM5PO started this project to try to predict those windows from
|
||
weather data. Graham W5ISP picked it up and is iterating on it from
|
||
there. The goal: a map that tells you when to get on the air, per
|
||
band, based on real atmospheric conditions and what we've learned
|
||
from the contacts you and everyone else have already made.
|
||
</p>
|
||
<div role="alert" class="alert alert-warning">
|
||
<p>
|
||
<strong>Heads up:</strong>
|
||
the scoring you see is hand-calibrated from the data we have so far.
|
||
It isn't updated in real time — I (Graham) re-fit the weights
|
||
manually when there's enough new data to move the needle. The more
|
||
contacts and beacon reports we collect, the better it gets. See the
|
||
<.link navigate={~p"/algo"} class="link link-hover font-semibold">algorithm page</.link>
|
||
for what's actually under the hood, and please <.link
|
||
navigate={~p"/submit"}
|
||
class="link link-hover font-semibold"
|
||
>submit your contacts</.link>.
|
||
</p>
|
||
</div>
|
||
<div role="alert" class="alert alert-info">
|
||
<p>
|
||
<strong>Coverage:</strong>
|
||
right now this focuses on the continental US. NOAA HRRR, ASOS,
|
||
IEMRE, and the ITU terrain data are CONUS-only. The Canadian HRDPS
|
||
model extends coverage north to 60°N (southern Canada), and SRTM
|
||
terrain tiles cover that range. Hawaii, Alaska, northern Canada,
|
||
and Mexico will need different data sources and aren't wired up yet.
|
||
</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it works, roughly</h2>
|
||
<p>
|
||
Two things, paired up. Most propagation tools keep them separate.
|
||
</p>
|
||
<ol class="list-decimal list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>The weather.</strong>
|
||
Hourly 3 km NOAA HRRR forecasts give surface fields and pressure-level
|
||
profiles across CONUS, while the Canadian HRDPS model (0.125°
|
||
resolution, 4× daily) extends coverage north to 60°N. We derive
|
||
refractivity profiles, minimum refractivity gradient, ducting
|
||
detection, boundary-layer depth, and precipitable water for every
|
||
grid cell. ASOS surface obs, 12-hourly upper-air soundings, and
|
||
gridded IEMRE reanalysis fill in the gaps. A 10-factor composite
|
||
score is written for every 0.125° cell (~95k across CONUS) across
|
||
23 microwave bands (902 MHz–241 GHz), for each hour of a 48-hour
|
||
forecast.
|
||
</li>
|
||
<li>
|
||
<strong>The contacts.</strong>
|
||
{format_count(@stats.contacts)}+ amateur microwave QSOs, each
|
||
tagged with the atmosphere at both ends of the path (and along it)
|
||
at the time the contact happened. That's a ground-truth dataset we
|
||
can actually fit against — "when you had this score, how far did
|
||
the contact go, and did it happen at all?"
|
||
</li>
|
||
</ol>
|
||
<p>
|
||
The scoring is a weighted sum of ten factors: rain, humidity,
|
||
precipitable water (PWAT), season, refractivity gradient, pressure,
|
||
T–Td depression, sky cover, wind, and time of day — weights
|
||
recalibrated via gradient descent against the contact dataset. The
|
||
physics changes by frequency: humidity helps at 10 GHz (more
|
||
refractivity) and hurts at 24+ GHz (absorption). Refractivity uses
|
||
native HRRR hybrid-sigma levels (10–50 m resolution) for duct
|
||
detection, with a pressure-level fallback (~250 m) when the native
|
||
data isn't available.
|
||
</p>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What we've collected so far</h2>
|
||
<p>Straight from the production database:</p>
|
||
<div class="grid grid-cols-2 md:grid-cols-3 gap-3">
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Contacts</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.contacts)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Weather stations</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.weather_stations)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Surface observations</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.surface_observations)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Upper-air soundings</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.soundings)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">HRRR profiles</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.hrrr_profiles)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">IEMRE gridded obs</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.iemre_observations)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Terrain profiles</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.terrain_profiles)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Beacons</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.beacons)}</div>
|
||
</div>
|
||
<div class="stat bg-base-200 rounded-box">
|
||
<div class="stat-title">Commercial link samples</div>
|
||
<div class="stat-value text-xl">{format_count(@stats.commercial_samples)}</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it's built</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
Elixir / Phoenix 1.8 + LiveView for everything web-facing. Ecto
|
||
on Postgres for storage. Oban runs the background pipelines
|
||
(HRRR pulls, terrain, ASOS, soundings, IEMRE, solar indices,
|
||
commercial links).
|
||
</li>
|
||
<li>
|
||
A Rust pipeline (prop-grid-rs) handles the heavy lifting:
|
||
fetching GRIB2 data from NOAA S3, decoding via wgrib2, deriving
|
||
weather scalars, and scoring all 23 bands in a single fused pass
|
||
over ~95k cells. Writes dense binary artifacts (.pgrid profiles,
|
||
.sgrid weather scalars, and per-band .prop score files) to NFS
|
||
for sub-millisecond single-cell reads from Elixir. A full
|
||
forecast-hour chain step runs in ~6 seconds — ~3× faster than
|
||
the previous Elixir-only pipeline.
|
||
</li>
|
||
<li>
|
||
Leaflet + Canvas tile layers for the propagation heatmap — we
|
||
render 0.125° cells at interactive frame rates. Tailwind v4 +
|
||
daisyUI for layout, esbuild for JS bundling.
|
||
</li>
|
||
<li>
|
||
Physics: ITU-R P.526-16 knife-edge + Deygout 3-edge terrain
|
||
diffraction, ITU-R P.838-3 rain attenuation, dynamic k-factor
|
||
from live HRRR refractivity gradients, great-circle geometry,
|
||
FSPL with frequency-dependent O₂ and H₂O absorption.
|
||
</li>
|
||
<li>
|
||
Data sources: NOAA HRRR (AWS S3, hourly analysis + 18 h
|
||
forecasts), Canadian HRDPS (MSC Datamart, 4× daily 48 h
|
||
forecasts), Iowa Environmental Mesonet (ASOS and upper-air
|
||
soundings), IEMRE gridded reanalysis, SRTM 90 m terrain, SNMP
|
||
polling on seven commercial microwave links near DFW at 5-minute
|
||
intervals, and NCEP NARR for pre-2014 contact enrichment.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What's next</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>Beacon calibration.</strong>
|
||
A distributed network of amateur receivers reporting CW beacon
|
||
signal levels on a schedule. Beacon submissions are already open
|
||
to anyone via the Beacons page.
|
||
</li>
|
||
<li>
|
||
<strong>Weight calibration.</strong>
|
||
Re-fit the 10 factor weights against recorded distances / QSO
|
||
counts per band so the score reflects real propagation, not hand
|
||
tuning. The PSKR spot ingestion pipeline (sampling every 5 min)
|
||
is building the calibration dataset automatically.
|
||
</li>
|
||
<li>
|
||
<strong>Better inputs.</strong> MRMS for precipitation at 24+ GHz where rain attenuation
|
||
dominates, RTMA/URMA surface analysis blending, GOES total
|
||
precipitable water.
|
||
</li>
|
||
<li>
|
||
<strong>Expanded coverage.</strong> The Canadian HRDPS model gives
|
||
us southern Canada (to 60°N). Alaska, Hawaii, Mexico, and the
|
||
Caribbean need their own data sources and scoring pipelines.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
<div role="alert" class="alert alert-info text-center">
|
||
<p>
|
||
If this tool is useful to you, consider
|
||
<a
|
||
href="https://www.paypal.com/ncp/payment/53VLBD2E67JAE"
|
||
target="_blank"
|
||
class="link font-semibold underline"
|
||
>
|
||
donating to NTMS
|
||
</a>
|
||
to help keep the project running.
|
||
</p>
|
||
</div>
|
||
</section>
|
||
</Layouts.app>
|
||
"""
|
||
end
|
||
|
||
# For tables that get huge (tens of millions of rows), using count(*) on
|
||
# every About page load is wasteful. We use PostgreSQL's pg_class.reltuples
|
||
# estimate — it's maintained by ANALYZE and is accurate to within a few
|
||
# percent, which is plenty for "big number on a marketing page."
|
||
@estimate_tables ~w(hrrr_profiles surface_observations)
|
||
|
||
defp fetch_stats do
|
||
%{
|
||
contacts: count_exact("contacts"),
|
||
weather_stations: count_exact("weather_stations"),
|
||
surface_observations: count_estimate("surface_observations"),
|
||
soundings: count_exact("soundings"),
|
||
hrrr_profiles: count_estimate("hrrr_profiles"),
|
||
iemre_observations: count_exact("iemre_observations"),
|
||
terrain_profiles: count_exact("terrain_profiles"),
|
||
beacons: count_exact("beacons"),
|
||
commercial_samples: count_exact("commercial_samples")
|
||
}
|
||
end
|
||
|
||
# Each count is wrapped individually so a missing table in dev (or a
|
||
# transient query failure) never blanks the entire dashboard. The
|
||
# previous outer try/rescue zeroed all 10 numbers if any single one
|
||
# raised — every stat read 0 on environments lacking hrrr_profiles
|
||
# or propagation_scores.
|
||
defp count_exact(table) do
|
||
case Repo.query("SELECT count(*) FROM #{table}") do
|
||
{:ok, %Postgrex.Result{rows: [[count]]}} -> count
|
||
_ -> 0
|
||
end
|
||
end
|
||
|
||
defp count_estimate(table) when table in @estimate_tables do
|
||
case Repo.query("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table]) do
|
||
{:ok, %Postgrex.Result{rows: [[estimate]]}} -> max(estimate || 0, 0)
|
||
_ -> 0
|
||
end
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000_000 do
|
||
"#{Float.round(n / 1_000_000, 1)}M"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000 do
|
||
"#{Float.round(n / 1_000, 1)}k"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n), do: Integer.to_string(n)
|
||
defp format_count(_), do: "—"
|
||
end
|