Graham McIntire
087517252b
Drops the formal "What we're trying to do / The approach / The stack" scaffolding for a shorter, first-person telling: line-of-sight is wrong, here's what we're doing about it, here's what's in the box, here's what's next. Adds a highlighted note that the scoring is hand-calibrated and not updated in real time, with links to the algorithm page and the submit page so readers can help close the loop.
242 lines
10 KiB
Elixir
242 lines
10 KiB
Elixir
defmodule MicrowavepropWeb.AboutLive do
|
||
@moduledoc false
|
||
use MicrowavepropWeb, :live_view
|
||
|
||
alias Microwaveprop.Repo
|
||
|
||
@impl true
|
||
def mount(_params, _session, socket) do
|
||
{:ok,
|
||
socket
|
||
|> assign(:page_title, "About")
|
||
|> assign(:stats, fetch_stats())}
|
||
end
|
||
|
||
@impl true
|
||
def render(assigns) do
|
||
~H"""
|
||
<Layouts.app flash={@flash} current_scope={@current_scope} max_width="max-w-4xl">
|
||
<.header>
|
||
About NTMS Propagation Prediction
|
||
<:subtitle>What we're building and why.</:subtitle>
|
||
</.header>
|
||
|
||
<section class="space-y-8 text-sm leading-relaxed">
|
||
<div class="space-y-3">
|
||
<p>
|
||
Line-of-sight is wrong for the microwave bands. Every operator on
|
||
902 MHz and up has made contacts over hills, through trees, and 200 km
|
||
past the horizon — contacts that shouldn't have worked if LOS were the
|
||
whole story. What's actually going on is the lower atmosphere:
|
||
temperature inversions, humidity gradients, ducting layers, and rain
|
||
cells that bend, trap, and absorb signals in ways that change by the
|
||
hour. The window for a good contact is often minutes, not days.
|
||
</p>
|
||
<p>
|
||
Jim KM5PO started this project to try to predict those windows from
|
||
weather data. Graham W5ISP picked it up and is iterating on it from
|
||
there. The goal: a map that tells you when to get on the air, per
|
||
band, based on real atmospheric conditions and what we've learned
|
||
from the contacts you and everyone else have already made.
|
||
</p>
|
||
<div class="rounded-box border border-warning/30 bg-warning/10 p-3">
|
||
<p>
|
||
<strong>Heads up:</strong>
|
||
the scoring you see is hand-calibrated from the data we have so far.
|
||
It isn't updated in real time — I (Graham) re-fit the weights
|
||
manually when there's enough new data to move the needle. The more
|
||
contacts and beacon reports we collect, the better it gets. See the
|
||
<.link navigate={~p"/algo"} class="link link-hover font-semibold">algorithm page</.link>
|
||
for what's actually under the hood, and please <.link
|
||
navigate={~p"/submit"}
|
||
class="link link-hover font-semibold"
|
||
>submit your contacts</.link>.
|
||
</p>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it works, roughly</h2>
|
||
<p>
|
||
Two things, paired up. Most propagation tools keep them separate.
|
||
</p>
|
||
<ol class="list-decimal list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>The weather.</strong>
|
||
Hourly 3 km HRRR forecasts give surface fields and pressure-level
|
||
profiles. We derive refractivity profiles, minimum refractivity
|
||
gradient, ducting detection, boundary-layer depth, and precipitable
|
||
water for every grid cell. ASOS surface obs, 12-hourly upper-air
|
||
soundings, and gridded IEMRE reanalysis fill in the gaps. A 9-factor
|
||
composite score is written out for every 0.125° cell across CONUS,
|
||
for each hour of an 18-hour forecast.
|
||
</li>
|
||
<li>
|
||
<strong>The contacts.</strong>
|
||
58k+ amateur microwave QSOs, each tagged with the atmosphere at
|
||
both ends of the path (and along it) at the time the contact
|
||
happened. That's a ground-truth dataset we can actually fit against
|
||
— "when you had this score, how far did the contact go, and did it
|
||
happen at all?"
|
||
</li>
|
||
</ol>
|
||
<p>
|
||
The scoring is a weighted sum of nine factors: humidity, time of
|
||
day, T–Td depression, refractivity gradient, sky cover, season, rain,
|
||
wind, pressure. The weights are band-dependent because the physics
|
||
changes as you go up: humidity helps at 10 GHz (more refractivity)
|
||
and hurts at 24+ GHz (absorption). Every coefficient is a hypothesis
|
||
I'm checking against the contact and beacon data.
|
||
</p>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What we've collected so far</h2>
|
||
<p>Straight from the production database:</p>
|
||
<div class="grid grid-cols-2 md:grid-cols-3 gap-3">
|
||
<.stat_card label="Contacts" value={@stats.contacts} />
|
||
<.stat_card label="Weather stations" value={@stats.weather_stations} />
|
||
<.stat_card label="Surface observations" value={@stats.surface_observations} />
|
||
<.stat_card label="Upper-air soundings" value={@stats.soundings} />
|
||
<.stat_card label="HRRR profiles" value={@stats.hrrr_profiles} />
|
||
<.stat_card label="IEMRE gridded obs" value={@stats.iemre_observations} />
|
||
<.stat_card label="Terrain profiles" value={@stats.terrain_profiles} />
|
||
<.stat_card label="Propagation scores" value={@stats.propagation_scores} />
|
||
<.stat_card label="Beacons" value={@stats.beacons} />
|
||
<.stat_card label="Commercial link samples" value={@stats.commercial_samples} />
|
||
</div>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">How it's built</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
Elixir / Phoenix 1.8 + LiveView for everything web-facing. Ecto
|
||
on Postgres for storage. Oban runs the background pipelines
|
||
(HRRR pulls, terrain, ASOS, soundings, IEMRE, solar indices,
|
||
commercial links).
|
||
</li>
|
||
<li>
|
||
Leaflet + Canvas tile layers for the propagation heatmap — we
|
||
render 0.125° cells at interactive frame rates. Tailwind v4 +
|
||
daisyUI for layout, esbuild for JS bundling.
|
||
</li>
|
||
<li>
|
||
Physics: ITU-R P.526-16 knife-edge + Deygout 3-edge terrain
|
||
diffraction, ITU-R P.838-3 rain attenuation, dynamic k-factor
|
||
from live HRRR refractivity gradients, great-circle geometry,
|
||
FSPL with frequency-dependent O₂ and H₂O absorption.
|
||
</li>
|
||
<li>
|
||
Data sources: NOAA HRRR (AWS S3, hourly analysis + 18 h
|
||
forecasts), Iowa Environmental Mesonet (ASOS and upper-air
|
||
soundings), IEMRE gridded reanalysis, SRTM 90 m terrain, SNMP
|
||
polling on seven commercial microwave links near DFW at 5-minute
|
||
intervals, and Copernicus ERA5 for pre-2014 contact enrichment.
|
||
</li>
|
||
<li>
|
||
There's Nx/Axon/EXLA scaffolding for a small feed-forward model
|
||
(13 features → 64 → 32 → 1 sigmoid). Not trained yet — waiting
|
||
on enough calibrated data to not overfit.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
|
||
<div class="space-y-3">
|
||
<h2 class="text-xl font-bold">What's next</h2>
|
||
<ul class="list-disc list-outside pl-5 space-y-2">
|
||
<li>
|
||
<strong>Beacon calibration.</strong>
|
||
A distributed network of amateur receivers reporting CW beacon
|
||
signal levels on a schedule. Beacon submissions are already open
|
||
to anyone via the Beacons page.
|
||
</li>
|
||
<li>
|
||
<strong>Weight calibration.</strong>
|
||
Fit the 9 factor weights against recorded distances / QSO counts
|
||
per band so the score reflects real propagation, not my
|
||
intuition.
|
||
</li>
|
||
<li>
|
||
<strong>Training the model.</strong>
|
||
Once there are enough clean (contact, conditions) pairs, train
|
||
the Axon model to augment or replace the hand-tuned scoring.
|
||
</li>
|
||
<li>
|
||
<strong>Better inputs.</strong> MRMS for precipitation at 24+ GHz where rain attenuation
|
||
dominates, RTMA/URMA surface analysis blending, GOES total
|
||
precipitable water.
|
||
</li>
|
||
</ul>
|
||
</div>
|
||
</section>
|
||
</Layouts.app>
|
||
"""
|
||
end
|
||
|
||
attr :label, :string, required: true
|
||
attr :value, :integer, required: true
|
||
|
||
defp stat_card(assigns) do
|
||
~H"""
|
||
<div class="rounded-lg border border-base-300 bg-base-200/40 p-3">
|
||
<div class="text-xs opacity-70 uppercase tracking-wide">{@label}</div>
|
||
<div class="text-xl font-bold tabular-nums">{format_count(@value)}</div>
|
||
</div>
|
||
"""
|
||
end
|
||
|
||
# For tables that get huge (tens of millions of rows), using count(*) on
|
||
# every About page load is wasteful. We use PostgreSQL's pg_class.reltuples
|
||
# estimate — it's maintained by ANALYZE and is accurate to within a few
|
||
# percent, which is plenty for "big number on a marketing page."
|
||
@estimate_tables ~w(hrrr_profiles propagation_scores surface_observations)
|
||
|
||
@stat_keys ~w(
|
||
contacts weather_stations surface_observations soundings hrrr_profiles
|
||
iemre_observations terrain_profiles propagation_scores beacons
|
||
commercial_samples
|
||
)a
|
||
|
||
defp fetch_stats do
|
||
%{
|
||
contacts: count_exact("contacts"),
|
||
weather_stations: count_exact("weather_stations"),
|
||
surface_observations: count_estimate("surface_observations"),
|
||
soundings: count_exact("soundings"),
|
||
hrrr_profiles: count_estimate("hrrr_profiles"),
|
||
iemre_observations: count_exact("iemre_observations"),
|
||
terrain_profiles: count_exact("terrain_profiles"),
|
||
propagation_scores: count_estimate("propagation_scores"),
|
||
beacons: count_exact("beacons"),
|
||
commercial_samples: count_exact("commercial_samples")
|
||
}
|
||
rescue
|
||
_ -> Map.new(@stat_keys, fn k -> {k, 0} end)
|
||
end
|
||
|
||
defp count_exact(table) do
|
||
%Postgrex.Result{rows: [[count]]} =
|
||
Repo.query!("SELECT count(*) FROM #{table}")
|
||
|
||
count
|
||
end
|
||
|
||
defp count_estimate(table) when table in @estimate_tables do
|
||
%Postgrex.Result{rows: [[estimate]]} =
|
||
Repo.query!("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table])
|
||
|
||
max(estimate || 0, 0)
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000_000 do
|
||
"#{Float.round(n / 1_000_000, 1)}M"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n) and n >= 1_000 do
|
||
"#{Float.round(n / 1_000, 1)}k"
|
||
end
|
||
|
||
defp format_count(n) when is_integer(n), do: Integer.to_string(n)
|
||
defp format_count(_), do: "—"
|
||
end
|