fix(about): per-table count fallback so missing tables don't blank everything

The whole-block try/rescue around fetch_stats meant any single failing
query (a missing dev table, a transient pg blip) zeroed all 10
counters. Each count_exact/1 + count_estimate/1 now uses Repo.query/2
and falls back to 0 only for its own column. Bumps the inline 9-factor
prose to 10-factor and pulls the contacts headline from the live count
instead of a stale '58k+' string.

Adds a CLAUDE.md note prompting future updates to refresh /about's
prose when factor counts, data sources, or schema scale change.
This commit is contained in:
Graham McIntire 2026-05-03 14:21:53 -05:00
parent 7ab2aa8a11
commit a0dfdd26af
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 32 additions and 29 deletions

View file

@ -221,6 +221,10 @@ User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers
- **Always run `cargo clippy --all-targets -- -D warnings` before committing Rust changes.** The Forgejo CI step runs clippy with warnings-as-errors and a passing local `cargo build` does NOT catch lints (e.g. `manual_range_contains`, `iter_kv_map`, `useless_format`). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the `Commands` section.
- The CI build pipeline runs `cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release`. Match this locally — `cargo build` alone is not enough.
### `/about` page maintenance
- Refresh `lib/microwaveprop_web/live/about_live.ex` whenever **algorithm structure**, **factor count or weights**, **major data sources**, **schema scale claims (e.g. "58k+ contacts")**, or **roadmap items** change. The dashboard counts auto-refresh from the DB; the prose blocks (overview, "How it works", "How it's built", "What's next") do not.
- The page renders a per-table count via `count_exact/1` and `count_estimate/1`. Each query is wrapped individually so a missing dev table doesn't blank the whole dashboard — keep that pattern when adding new stats.
### Deployment
- Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...`
- Deployment name `prop` (3 replicas). Inspect with `kubectl -n prop get pods`, `kubectl -n prop logs deploy/prop`, `kubectl -n prop exec deploy/prop -- ...`

View file

@ -76,27 +76,29 @@ defmodule MicrowavepropWeb.AboutLive do
profiles. We derive refractivity profiles, minimum refractivity
gradient, ducting detection, boundary-layer depth, and precipitable
water for every grid cell. ASOS surface obs, 12-hourly upper-air
soundings, and gridded IEMRE reanalysis fill in the gaps. A 9-factor
composite score is written out for every 0.125° cell across CONUS,
for each hour of an 18-hour forecast.
soundings, and gridded IEMRE reanalysis fill in the gaps. A
10-factor composite score is written out for every 0.125° cell
across CONUS, for each hour of an 18-hour forecast.
</li>
<li>
<strong>The contacts.</strong>
58k+ amateur microwave QSOs, each tagged with the atmosphere at
both ends of the path (and along it) at the time the contact
happened. That's a ground-truth dataset we can actually fit against
"when you had this score, how far did the contact go, and did it
happen at all?"
{format_count(@stats.contacts)}+ amateur microwave QSOs, each
tagged with the atmosphere at both ends of the path (and along it)
at the time the contact happened. That's a ground-truth dataset we
can actually fit against "when you had this score, how far did
the contact go, and did it happen at all?"
</li>
</ol>
<p>
The scoring is a weighted sum of ten factors: rain, humidity,
precipitable water (PWAT), season, refractivity gradient, pressure,
TTd depression, sky cover, wind, and time of day. Weights were
calibrated via gradient descent against 5,000 QSOs. The physics
changes by frequency: humidity helps at 10 GHz (more refractivity)
and hurts at 24+ GHz (absorption). Refractivity now uses native
HRRR hybrid-sigma levels (1050 m resolution) for duct detection.
TTd depression, sky cover, wind, and time of day weights
recalibrated via gradient descent against the contact dataset. The
physics changes by frequency: humidity helps at 10 GHz (more
refractivity) and hurts at 24+ GHz (absorption). Refractivity uses
native HRRR hybrid-sigma levels (1050 m resolution) for duct
detection, with a pressure-level fallback (~250 m) when the native
data isn't available.
</p>
</div>
@ -233,12 +235,6 @@ defmodule MicrowavepropWeb.AboutLive do
# percent, which is plenty for "big number on a marketing page."
@estimate_tables ~w(hrrr_profiles propagation_scores surface_observations)
@stat_keys ~w(
contacts weather_stations surface_observations soundings hrrr_profiles
iemre_observations terrain_profiles propagation_scores beacons
commercial_samples
)a
defp fetch_stats do
%{
contacts: count_exact("contacts"),
@ -252,22 +248,25 @@ defmodule MicrowavepropWeb.AboutLive do
beacons: count_exact("beacons"),
commercial_samples: count_exact("commercial_samples")
}
rescue
_ -> Map.new(@stat_keys, fn k -> {k, 0} end)
end
# Each count is wrapped individually so a missing table in dev (or a
# transient query failure) never blanks the entire dashboard. The
# previous outer try/rescue zeroed all 10 numbers if any single one
# raised — every stat read 0 on environments lacking hrrr_profiles
# or propagation_scores.
defp count_exact(table) do
%Postgrex.Result{rows: [[count]]} =
Repo.query!("SELECT count(*) FROM #{table}")
count
case Repo.query("SELECT count(*) FROM #{table}") do
{:ok, %Postgrex.Result{rows: [[count]]}} -> count
_ -> 0
end
end
defp count_estimate(table) when table in @estimate_tables do
%Postgrex.Result{rows: [[estimate]]} =
Repo.query!("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table])
max(estimate || 0, 0)
case Repo.query("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table]) do
{:ok, %Postgrex.Result{rows: [[estimate]]}} -> max(estimate || 0, 0)
_ -> 0
end
end
defp format_count(n) when is_integer(n) and n >= 1_000_000 do