243 lines
10 KiB
Elixir
243 lines
10 KiB
Elixir
defmodule MicrowavepropWeb.AboutLive do
|
||
@moduledoc false
|
||
use MicrowavepropWeb, :live_view
|
||
|
||
alias Microwaveprop.Repo
|
||
|
||
@impl true
|
||
def mount(_params, _session, socket) do
|
||
{:ok,
|
||
socket
|
||
|> assign(:page_title, "About")
|
||
|> assign(:stats, fetch_stats())}
|
||
end
|
||
|
||
@impl true
|
||
def render(assigns) do
|
||
~H"""
|
||
<Layouts.app flash={@flash} current_scope={@current_scope} max_width="max-w-4xl">
|
||
<.header>
|
||
About NTMS Propagation Prediction
|
||
<:subtitle>What we're building and why.</:subtitle>
|
||
</.header>
|
||
|
||
<section class="space-y-8 text-sm leading-relaxed">
|
||
<div class="space-y-3">
|
||
<p>
|
||
Line-of-sight is wrong for the microwave bands. Every operator on
|
||
902 MHz and up has made contacts over hills, through trees, and 200 km
|
||
past the horizon — contacts that shouldn't have worked if LOS were the
|
||
whole story. What's actually going on is the lower atmosphere:
|
||
temperature inversions, humidity gradients, ducting layers, and rain
|
||
cells that bend, trap, and absorb signals in ways that change by the
|
||
hour. The window for a good contact is often minutes, not days.
|
||
</p>
|
||
<p>
|
||
Jim KM5PO started this project to try to predict those windows from
|
||
weather data. Graham W5ISP picked it up and is iterating on it from
|
||
there. The goal: a map that tells you when to get on the air, per
|
||
band, based on real atmospheric conditions and what we've learned
|
||
from the contacts you and everyone else have already made.
|
||
</p>
|
||
<div class="rounded-box border border-warning/30 bg-warning/10 p-3">
|
||
<p>
|
||
<strong>Heads up:</strong>
|
||
the scoring you see is hand-calibrated from the data we have so far.
|
||
It isn't updated in real time — I (Graham) re-fit the weights
|
||
manually when there's enough new data to move the needle. The more
|
||
contacts and beacon reports we collect, the better it gets. See the
|
||
<.link navigate={~p"/algo"} class="link link-hover font-semibold">algorithm page</.link>
|
||
for what's actually under the hood, and please <.link
|
||
navigate={~p"/submit"}
|
||
class="link link-hover font-semibold"
|
||
>submit your contacts</.link>.
|
||
</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it works, roughly</h2>
|
||
<p>
|
||
Two things, paired up. Most propagation tools keep them separate.
|
||
</p>
|
||
<ol class="list-decimal list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>The weather.</strong>
|
||
Hourly 3 km HRRR forecasts give surface fields and pressure-level
|
||
profiles. We derive refractivity profiles, minimum refractivity
|
||
gradient, ducting detection, boundary-layer depth, and precipitable
|
||
water for every grid cell. ASOS surface obs, 12-hourly upper-air
|
||
soundings, and gridded IEMRE reanalysis fill in the gaps. A 9-factor
|
||
composite score is written out for every 0.125° cell across CONUS,
|
||
for each hour of an 18-hour forecast.
|
||
</li>
|
||
<li>
|
||
<strong>The contacts.</strong>
|
||
58k+ amateur microwave QSOs, each tagged with the atmosphere at
|
||
both ends of the path (and along it) at the time the contact
|
||
happened. That's a ground-truth dataset we can actually fit against
|
||
— "when you had this score, how far did the contact go, and did it
|
||
happen at all?"
|
||
</li>
|
||
</ol>
|
||
<p>
|
||
The scoring is a weighted sum of ten factors: rain, humidity,
|
||
precipitable water (PWAT), season, refractivity gradient, pressure,
|
||
T–Td depression, sky cover, wind, and time of day. Weights were
|
||
calibrated via gradient descent against 5,000 QSOs. The physics
|
||
changes by frequency: humidity helps at 10 GHz (more refractivity)
|
||
and hurts at 24+ GHz (absorption). Refractivity now uses native
|
||
HRRR hybrid-sigma levels (10–50 m resolution) for duct detection.
|
||
</p>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What we've collected so far</h2>
|
||
<p>Straight from the production database:</p>
|
||
<div class="grid grid-cols-2 md:grid-cols-3 gap-3">
|
||
<.stat_card label="Contacts" value={@stats.contacts} />
|
||
<.stat_card label="Weather stations" value={@stats.weather_stations} />
|
||
<.stat_card label="Surface observations" value={@stats.surface_observations} />
|
||
<.stat_card label="Upper-air soundings" value={@stats.soundings} />
|
||
<.stat_card label="HRRR profiles" value={@stats.hrrr_profiles} />
|
||
<.stat_card label="IEMRE gridded obs" value={@stats.iemre_observations} />
|
||
<.stat_card label="Terrain profiles" value={@stats.terrain_profiles} />
|
||
<.stat_card label="Propagation scores" value={@stats.propagation_scores} />
|
||
<.stat_card label="Beacons" value={@stats.beacons} />
|
||
<.stat_card label="Commercial link samples" value={@stats.commercial_samples} />
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it's built</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
Elixir / Phoenix 1.8 + LiveView for everything web-facing. Ecto
|
||
on Postgres for storage. Oban runs the background pipelines
|
||
(HRRR pulls, terrain, ASOS, soundings, IEMRE, solar indices,
|
||
commercial links).
|
||
</li>
|
||
<li>
|
||
Leaflet + Canvas tile layers for the propagation heatmap — we
|
||
render 0.125° cells at interactive frame rates. Tailwind v4 +
|
||
daisyUI for layout, esbuild for JS bundling.
|
||
</li>
|
||
<li>
|
||
Physics: ITU-R P.526-16 knife-edge + Deygout 3-edge terrain
|
||
diffraction, ITU-R P.838-3 rain attenuation, dynamic k-factor
|
||
from live HRRR refractivity gradients, great-circle geometry,
|
||
FSPL with frequency-dependent O₂ and H₂O absorption.
|
||
</li>
|
||
<li>
|
||
Data sources: NOAA HRRR (AWS S3, hourly analysis + 18 h
|
||
forecasts), Iowa Environmental Mesonet (ASOS and upper-air
|
||
soundings), IEMRE gridded reanalysis, SRTM 90 m terrain, SNMP
|
||
polling on seven commercial microwave links near DFW at 5-minute
|
||
intervals, and Copernicus ERA5 for pre-2014 contact enrichment.
|
||
</li>
|
||
<li>
|
||
There's Nx/Axon/EXLA scaffolding for a small feed-forward model
|
||
(13 features → 64 → 32 → 1 sigmoid). Not trained yet — waiting
|
||
on enough calibrated data to not overfit.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What's next</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>Beacon calibration.</strong>
|
||
A distributed network of amateur receivers reporting CW beacon
|
||
signal levels on a schedule. Beacon submissions are already open
|
||
to anyone via the Beacons page.
|
||
</li>
|
||
<li>
|
||
<strong>Weight calibration.</strong>
|
||
Fit the 9 factor weights against recorded distances / QSO counts
|
||
per band so the score reflects real propagation, not my
|
||
intuition.
|
||
</li>
|
||
<li>
|
||
<strong>Training the model.</strong>
|
||
Once there are enough clean (contact, conditions) pairs, train
|
||
the Axon model to augment or replace the hand-tuned scoring.
|
||
</li>
|
||
<li>
|
||
<strong>Better inputs.</strong> MRMS for precipitation at 24+ GHz where rain attenuation
|
||
dominates, RTMA/URMA surface analysis blending, GOES total
|
||
precipitable water.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</section>
|
||
</Layouts.app>
|
||
"""
|
||
end
|
||
|
||
attr :label, :string, required: true
|
||
attr :value, :integer, required: true
|
||
|
||
defp stat_card(assigns) do
|
||
~H"""
|
||
<div class="rounded-lg border border-base-300 bg-base-200/40 p-3">
|
||
<div class="text-xs opacity-70 uppercase tracking-wide">{@label}</div>
|
||
<div class="text-xl font-bold tabular-nums">{format_count(@value)}</div>
|
||
</div>
|
||
"""
|
||
end
|
||
|
||
# For tables that get huge (tens of millions of rows), using count(*) on
|
||
# every About page load is wasteful. We use PostgreSQL's pg_class.reltuples
|
||
# estimate — it's maintained by ANALYZE and is accurate to within a few
|
||
# percent, which is plenty for "big number on a marketing page."
|
||
@estimate_tables ~w(hrrr_profiles propagation_scores surface_observations)
|
||
|
||
@stat_keys ~w(
|
||
contacts weather_stations surface_observations soundings hrrr_profiles
|
||
iemre_observations terrain_profiles propagation_scores beacons
|
||
commercial_samples
|
||
)a
|
||
|
||
defp fetch_stats do
|
||
%{
|
||
contacts: count_exact("contacts"),
|
||
weather_stations: count_exact("weather_stations"),
|
||
surface_observations: count_estimate("surface_observations"),
|
||
soundings: count_exact("soundings"),
|
||
hrrr_profiles: count_estimate("hrrr_profiles"),
|
||
iemre_observations: count_exact("iemre_observations"),
|
||
terrain_profiles: count_exact("terrain_profiles"),
|
||
propagation_scores: count_estimate("propagation_scores"),
|
||
beacons: count_exact("beacons"),
|
||
commercial_samples: count_exact("commercial_samples")
|
||
}
|
||
rescue
|
||
_ -> Map.new(@stat_keys, fn k -> {k, 0} end)
|
||
end
|
||
|
||
defp count_exact(table) do
|
||
%Postgrex.Result{rows: [[count]]} =
|
||
Repo.query!("SELECT count(*) FROM #{table}")
|
||
|
||
count
|
||
end
|
||
|
||
defp count_estimate(table) when table in @estimate_tables do
|
||
%Postgrex.Result{rows: [[estimate]]} =
|
||
Repo.query!("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table])
|
||
|
||
max(estimate || 0, 0)
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000_000 do
|
||
"#{Float.round(n / 1_000_000, 1)}M"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000 do
|
||
"#{Float.round(n / 1_000, 1)}k"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n), do: Integer.to_string(n)
|
||
defp format_count(_), do: "—"
|
||
end
|