fix(about): per-table count fallback so missing tables don't blank everything

The whole-block try/rescue around fetch_stats meant any single failing
query (a missing dev table, a transient pg blip) zeroed all 10
counters. Each count_exact/1 + count_estimate/1 now uses Repo.query/2
and falls back to 0 only for its own column. Bumps the inline 9-factor
prose to 10-factor and pulls the contacts headline from the live count
instead of a stale '58k+' string.

Adds a CLAUDE.md note prompting future updates to refresh /about's
prose when factor counts, data sources, or schema scale change.
This commit is contained in:
Graham McIntire 2026-05-03 14:21:53 -05:00
parent 7ab2aa8a11
commit a0dfdd26af
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 32 additions and 29 deletions

View file

@ -221,6 +221,10 @@ User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers
- **Always run `cargo clippy --all-targets -- -D warnings` before committing Rust changes.** The Forgejo CI step runs clippy with warnings-as-errors and a passing local `cargo build` does NOT catch lints (e.g. `manual_range_contains`, `iter_kv_map`, `useless_format`). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the `Commands` section. - **Always run `cargo clippy --all-targets -- -D warnings` before committing Rust changes.** The Forgejo CI step runs clippy with warnings-as-errors and a passing local `cargo build` does NOT catch lints (e.g. `manual_range_contains`, `iter_kv_map`, `useless_format`). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the `Commands` section.
- The CI build pipeline runs `cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release`. Match this locally — `cargo build` alone is not enough. - The CI build pipeline runs `cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release`. Match this locally — `cargo build` alone is not enough.
### `/about` page maintenance
- Refresh `lib/microwaveprop_web/live/about_live.ex` whenever **algorithm structure**, **factor count or weights**, **major data sources**, **schema scale claims (e.g. "58k+ contacts")**, or **roadmap items** change. The dashboard counts auto-refresh from the DB; the prose blocks (overview, "How it works", "How it's built", "What's next") do not.
- The page renders a per-table count via `count_exact/1` and `count_estimate/1`. Each query is wrapped individually so a missing dev table doesn't blank the whole dashboard — keep that pattern when adding new stats.
### Deployment ### Deployment
- Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...` - Production runs on Kubernetes (`home-cluster` context, `prop` namespace) — use `kubectl -n prop ...`
- Deployment name `prop` (3 replicas). Inspect with `kubectl -n prop get pods`, `kubectl -n prop logs deploy/prop`, `kubectl -n prop exec deploy/prop -- ...` - Deployment name `prop` (3 replicas). Inspect with `kubectl -n prop get pods`, `kubectl -n prop logs deploy/prop`, `kubectl -n prop exec deploy/prop -- ...`

View file

@ -76,27 +76,29 @@ defmodule MicrowavepropWeb.AboutLive do
profiles. We derive refractivity profiles, minimum refractivity profiles. We derive refractivity profiles, minimum refractivity
gradient, ducting detection, boundary-layer depth, and precipitable gradient, ducting detection, boundary-layer depth, and precipitable
water for every grid cell. ASOS surface obs, 12-hourly upper-air water for every grid cell. ASOS surface obs, 12-hourly upper-air
soundings, and gridded IEMRE reanalysis fill in the gaps. A 9-factor soundings, and gridded IEMRE reanalysis fill in the gaps. A
composite score is written out for every 0.125° cell across CONUS, 10-factor composite score is written out for every 0.125° cell
for each hour of an 18-hour forecast. across CONUS, for each hour of an 18-hour forecast.
</li> </li>
<li> <li>
<strong>The contacts.</strong> <strong>The contacts.</strong>
58k+ amateur microwave QSOs, each tagged with the atmosphere at {format_count(@stats.contacts)}+ amateur microwave QSOs, each
both ends of the path (and along it) at the time the contact tagged with the atmosphere at both ends of the path (and along it)
happened. That's a ground-truth dataset we can actually fit against at the time the contact happened. That's a ground-truth dataset we
"when you had this score, how far did the contact go, and did it can actually fit against "when you had this score, how far did
happen at all?" the contact go, and did it happen at all?"
</li> </li>
</ol> </ol>
<p> <p>
The scoring is a weighted sum of ten factors: rain, humidity, The scoring is a weighted sum of ten factors: rain, humidity,
precipitable water (PWAT), season, refractivity gradient, pressure, precipitable water (PWAT), season, refractivity gradient, pressure,
TTd depression, sky cover, wind, and time of day. Weights were TTd depression, sky cover, wind, and time of day weights
calibrated via gradient descent against 5,000 QSOs. The physics recalibrated via gradient descent against the contact dataset. The
changes by frequency: humidity helps at 10 GHz (more refractivity) physics changes by frequency: humidity helps at 10 GHz (more
and hurts at 24+ GHz (absorption). Refractivity now uses native refractivity) and hurts at 24+ GHz (absorption). Refractivity uses
HRRR hybrid-sigma levels (1050 m resolution) for duct detection. native HRRR hybrid-sigma levels (1050 m resolution) for duct
detection, with a pressure-level fallback (~250 m) when the native
data isn't available.
</p> </p>
</div> </div>
@ -233,12 +235,6 @@ defmodule MicrowavepropWeb.AboutLive do
# percent, which is plenty for "big number on a marketing page." # percent, which is plenty for "big number on a marketing page."
@estimate_tables ~w(hrrr_profiles propagation_scores surface_observations) @estimate_tables ~w(hrrr_profiles propagation_scores surface_observations)
@stat_keys ~w(
contacts weather_stations surface_observations soundings hrrr_profiles
iemre_observations terrain_profiles propagation_scores beacons
commercial_samples
)a
defp fetch_stats do defp fetch_stats do
%{ %{
contacts: count_exact("contacts"), contacts: count_exact("contacts"),
@ -252,22 +248,25 @@ defmodule MicrowavepropWeb.AboutLive do
beacons: count_exact("beacons"), beacons: count_exact("beacons"),
commercial_samples: count_exact("commercial_samples") commercial_samples: count_exact("commercial_samples")
} }
rescue
_ -> Map.new(@stat_keys, fn k -> {k, 0} end)
end end
# Each count is wrapped individually so a missing table in dev (or a
# transient query failure) never blanks the entire dashboard. The
# previous outer try/rescue zeroed all 10 numbers if any single one
# raised — every stat read 0 on environments lacking hrrr_profiles
# or propagation_scores.
defp count_exact(table) do defp count_exact(table) do
%Postgrex.Result{rows: [[count]]} = case Repo.query("SELECT count(*) FROM #{table}") do
Repo.query!("SELECT count(*) FROM #{table}") {:ok, %Postgrex.Result{rows: [[count]]}} -> count
_ -> 0
count end
end end
defp count_estimate(table) when table in @estimate_tables do defp count_estimate(table) when table in @estimate_tables do
%Postgrex.Result{rows: [[estimate]]} = case Repo.query("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table]) do
Repo.query!("SELECT reltuples::bigint FROM pg_class WHERE relname = $1", [table]) {:ok, %Postgrex.Result{rows: [[estimate]]}} -> max(estimate || 0, 0)
_ -> 0
max(estimate || 0, 0) end
end end
defp format_count(n) when is_integer(n) and n >= 1_000_000 do defp format_count(n) when is_integer(n) and n >= 1_000_000 do