Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
- Unify haversine_km / deg_to_rad: Geo is the canonical home (atan2
form for antipodal stability); Radio, common_volume, rain_scatter,
ionosphere, terrain/srtm, terrain/elevation_client now delegate.
- Drop safe_min/safe_max wrappers in favor of Enum.min/max defaults.
- Move parse_float / parse_int to MicrowavepropWeb.LiveHelpers; eme,
path, and rover LiveViews import from there.
- Extract MicrowavepropWeb.LocationResolver from the eme/path
resolve_source / resolve_location duplicate. Standardize the kind
key and "Invalid grid square" error message.
- Convert Scorer cond chains (humidity, td_depression, sky, wind,
rain, pwat, pressure) to multi-clause guarded defs, matching the
existing classify_time_period style.
- extract_profile_fields uses a named map accumulator instead of a
6-tuple; build_path_conditions guards on %{temps: []} / %{dewpoints: []}.
- Collapse scores_file clamp/3 to max |> min.
- humidity_effect_label lives in BandConfig; map_live and path_live
both use it.
Sweep of remaining swallowed-error sites flagged by an audit pass:
- weather/gefs_client.ex: byte-range download task exits log url + reason
before being surfaced as {:error, _} (matches the HRRR fix).
- live/path_live.ex: terrain-profile load failure, native-duct lookup
failure, and ProfilesFile.read failure now log warnings with path
endpoints / midpoint / valid_time. Previously rendered a degraded
result silently.
- live/rover_live.ex: station-mutation {:error, _} now logs id + reason
instead of dropping into {:noreply, socket}.
- application.ex: the boot-time backfill_missing_home_qth Task.start is
now wrapped in try/rescue. A DB issue at boot would have crashed the
task silently (no Logger handler attached pre-Logger crash format).
The radar-only RainScatterClassifier sees ducting only when HRRR's
pressure-level M-profile flagged it. Sounding-detected and gradient-
inferred ducts surface in elevation_profile.ducts via the analysis
pipeline — promote those into :tropo_duct so the Mechanism badge
agrees with the narrative.
Click a row's info area to expand a Leaflet mini-map centered on that
pin, defaulting to ESRI World_Imagery satellite (zoom up to 21) with
OSM/Topo toggles in a top-right layer control.
Adds a checkbox in the rover sidebar that constrains the suggested
candidate spots to lat/lon points tagged as :ideal in /rover-locations.
When enabled, Compute.run replaces grid cells with synthetic cells at
each ideal location's exact coordinates, inheriting the propagation
score from the nearest grid cell so atmospheric forecast still factors
in. Path-clearance, terrain, building, and canopy penalties run
unchanged on those snapped candidates.
New globally-shared list of rover-friendly (or off-limits) parking
spots. Anyone can view; logged-in users can add new locations and
edit/delete their own. Each location stores lat/lon, free-text notes,
and a status enum (:ideal | :off_limits).
* Microwaveprop.Rover.Location schema + migration
* Rover.list_locations/0, create_location/2, update_location/3,
delete_location/2 (mutations require User; ownership-scoped)
* /rover-locations LiveView in the public live_session — Add button
is disabled for anonymous visitors, Edit/Delete buttons render
only on the user's own entries
* Tests for context + LiveView covering anonymous read, owner-only
edit/delete, and form submission
Adds Microwaveprop.Canopy module that reads 1° × 1° uint8 per-degree
canopy-height tiles (mirrors SRTM .hgt naming/layout, 30 m resolution).
Returns 0 m for areas with no tile on disk so the lookup is safe to
thread through the existing path-clearance pipeline before any data
lands.
Wires canopy into:
- PathTerrain.obstacle_top: per-sample blocker = ground +
max(building_m, canopy_m), so the rover ranks paths through forests
as obstructed even when no buildings sit on the line
- CandidateDetail profile: each sample now carries canopy_m alongside
building_m
- Rover-detail SVG: green "trees" polygon stacked on terrain, under
the red building polygon
- Path calculator elevation chart: green Tree Canopy dataset between
terrain and buildings
Data prep (download Potapov 2019 / Lang 2022 GeoTIFF, slice to per-degree
uint8 tiles, drop in /data/canopy/) is a separate one-shot operation —
without tiles, lookups return 0 and the existing behavior is unchanged.
Path calculator now queries Microsoft Building Footprints for the
tallest structure within 80m of each profile sample and renders a red
"Buildings" dataset on top of the terrain layer in the elevation chart.
Also normalize on-disk grid HRRR cells to include the legacy
:min_refractivity_gradient / :surface_refractivity / :ducting_detected
keys (mapped from :native_min_gradient and :duct_count). Path-calculator
crashed in prod with KeyError when a path's HRRR points came from the
new on-disk grid format instead of the DB profile shape.
Compute.run now accepts a `progress` callback and reports human-readable
labels before each pipeline step (loading grid, looking up elevation,
checking road access, scoring cells, etc.). RoverLive feeds it via
send/2 to itself and renders the latest label as small text to the left
of the Calculate button while it's running.
Each profile sample now carries the tallest building height within 80m
of the path point. The candidate-detail SVG renders these as a red
polygon stacked on top of the terrain so blockage from buildings is
visible alongside ridges. max_obstacle_m now reflects building tops too,
so the clearance label downgrades when buildings sit on the link.
Phase 2/3/5 of the building-blockage support:
* Parser streams csv.gz tiles into compact records (centroid, radius,
height) — keeps RAM ~32 B per polygon, drops -1.0-height entries.
* Index buckets records into 0.01° (~1 km) grid cells in a public ETS
table for concurrent reads; max_height_near/3 and records_near/3
scan only the 9 nearest buckets.
* Loader lazily parses any cached quadkey before a Calculate so the
scorer is terrain-only when tiles aren't on disk yet.
* Rover.PathTerrain now treats each path-sample obstacle as
terrain_elev + max_local_building_height, so blocked LOS through
recent construction actually reduces clearance.
* Selecting a candidate pushes a candidate_buildings event with the
buildings near each rover→station path; the hook renders them as
height-coloured circles (yellow <10 m, orange 10-30 m, red >=30 m)
with a tooltip showing height in meters.
- Detail panel now reads as an aiming card: "Aim 220° (SW) · clearance +12 ft"
with the precise bearing in degrees, plus rover/obstacle elev on a second line
- Elevation profile labels swapped from min/max-of-profile to start (rover) and
end (station) elevations, since their positions in the chart now match
- Map draws a dashed indigo polyline from the candidate to each selected
station when a candidate is opened; cleared when the panel closes
The /path calculator showed "0 / 9 HRRR points" in production despite
the on-disk profile store being current. Root cause: profile_from_cell/2
treated the cell's :profile key as a wrapper sub-map and called
Map.put_new on it — but the :profile key actually holds the vertical
pressure-level LIST. Every point sample crashed with BadMapError, the
crash propagated as {:exit, _} through Task.async_stream, and the
consumer silently dropped all 9 results.
Fix: stop wrapping. Cells are already flat HrrrProfile-shaped maps;
just stamp lat/lon (from the caller, since cells don't carry their
own coords — those are the map key) and valid_time onto the cell.
Audit + log every other async error path so the next silent failure
isn't invisible:
- PathLive HRRR point lookup
- Propagation.point_forecast per-hour reads
- Viewshed ray crashes
- IemClient ASOS network fetches
- RtmaClient range-download tasks
- Recalibrator factor-vector batches (positive + negative samples)
- MapLive forecast preload tasks
- RoverLive station resolution
LiveViews already had handle_async/3 exit clauses with logging. The
gap was always in Task.async_stream consumers that wrote {:exit, _} -> []
without surfacing the reason.
Add the rule to CLAUDE.md and project memory so this never repeats.
Also fix a pre-existing skewt_svg.ex compiler warning where
@critical_label_min_dy was used before being defined.
LCL/LFC and 0 °C/WBZ landed within ~5 px of each other on the right-
edge label rail and stomped on each other's text. Sort by y, bump
each subsequent label down by at least 14 px from the previous one,
and draw a thin leader from the true tick to the bumped baseline so
nothing reads as misaligned.
Also widened the SVG from 720 → 820 user units so longer entries
like `LCL 925 mb` no longer get clipped by the right edge.
The 18-hour forecast chart on /path now shows the same weighted-
criteria panel the /map popup builds when the user clicks a grid
cell. Hovering a dot fires a server roundtrip that fetches
Propagation.point_detail at the path's midpoint for that valid_time;
results are cached client-side so re-hovering is instant.
`Propagation.point_detail/4` now returns a `:profile_source` tag —
`:exact`, `{:fallback, fallback_valid_time}`, or `:unavailable` —
distinguishing real per-hour breakdowns (Rust pipeline writes a
profile file for every f00..f18 step) from rare cycle-miss fallbacks
where we approximate from a recent analysis profile. Stale comment
about "only f00 persists profiles" updated.
Map utilities (FACTOR_META, FACTOR_ORDER, factorBar,
factorExplanation, scoreTier, FactorMeta, ScoreTier) are now exported
from propagation_map_hook so the new path hook reuses the exact same
look + copy.
The summary card's "0 / 9 points" came from `Weather.find_nearest_hrrr`
only checking the `hrrr_profiles` DB table, which is populated solely
by per-QSO enrichment within ±0.07° / ±1 h. Arbitrary paths off any
recently-enriched contact got nothing.
Switch to ProfilesFile.read_point as the primary source — the hourly
chain publishes full-CONUS f00..f18 cells there — and keep the
DB-table query as the fallback when the on-disk store is empty or
doesn't cover a point. Cell shape is flattened so the template and
build_scoring/6 stay shape-agnostic.
The grid-cached profile is trimmed to 13 levels (1000→700 mb) for
memory reasons, so the skew-T plot showed nothing above 700 mb.
HrrrClient.fetch_profile/4 now accepts a forecast_hour opt; SkewtLive
issues a per-point fetch against the same cycle the cached row came
from to get the full 1000→100 mb set, falling back to the cached
truncated cell on any failure.
Time selection no longer round-trips through the URL — `select_time`
updates state in-process and triggers a focused :load_skewt_time
async, so refreshing or sharing a /skewt?q=... link always lands on
the most recent analysis hour.
URL-parameter loads (`/skewt?q=…`) were blocking the initial mount
on three serial steps: geocoder/grid resolve, ProfilesFile or
HrrrProfileLookup read, and the SkewtParams/SoundingParams/SkewtSvg
derivation. The address path in particular hits an external API,
which made the page take "forever" to load on first paint.
Restructure SkewtLive so the page chrome (header, search bar) renders
synchronously on mount, and the heavy work runs in a single
`start_async(:load_skewt, …)` task that calls SkewtLocationResolver,
walks the on-disk profile store with a DB fallback, and produces the
SVG. A new `loading?` assign drives a small spinner + "Resolving
location and loading HRRR profile…" message that shows while the task
is in flight; previous data stays on screen so navigation between
forecast hours doesn't blank out the diagram.
handle_async/3 covers three landings:
* `{:ok, {:ok, result}}` — apply assigns from the resolver
* `{:ok, {:error, reason}}` — surface as the existing error alert
(geocoder failure, unknown callsign, etc.)
* `{:exit, reason}` — log a warning and show a generic retry message
start_async/3 cancels any prior task with the same name on each call,
so a rapid-fire URL patch (e.g. clicking through forecast-hour
buttons before the first response returns) never lands stale data.
Tests: extended SkewtLiveTest from 3 to 4 cases — the URL-param
initial render now asserts the chrome + spinner appear pre-async,
and a separate case `render_async/2`s the view to confirm the
location label arrives once the task completes. 4/4 green.
Three additions that bring the diagram closer to the SPC sounding
viewer's full-information chart:
* Wet-bulb temperature trace, plotted as a thinner blue line under
the dewpoint trace. Comes from a Stipanuk-style bracket between
the level's dewpoint and dry-bulb temperatures (already used by
`SkewtParams` for the wet-bulb-zero level — now public so the
renderer can call it per level).
* Environment virtual-temperature trace and parcel virtual-T
trajectory, both as thin pinkish dashed lines. The parcel
virtual-T is the actual buoyancy curve CAPE integrates against,
so showing it next to the dry-bulb parcel ascent makes the
moisture correction visible. New `SkewtParams.virtual_temp_c/2`
and `/3` helpers, plus `SkewtParams.mixing_ratio/2`.
* Critical-level rail on the right edge. Each of LCL / LFC / EL /
0 °C / WBZ that resolves on the current profile gets a short
horizontal tick at the right of the plot and a `<label> <p> mb`
tag just outside it — matches the SPC chart's `143 mb (mix)` /
`158 mb (mix)` annotation column. SkewtLive computes the
pressures (and converts the height-keyed 0 °C and WBZ from
SkewtParams back to pressure via `pressure_at_height/2`) before
handing them to `SkewtSvg.render/2`.
Wind-derived overlays (hodograph, wind barbs) remain off — HRRR
profile data is wind-free above 10 m AGL.
mix test: 27 / 27 across the four /skewt-related test files,
2,902 / 2,902 + 221 properties on the full suite. mix credo --strict
clean (one cyclomatic-complexity refactor extracted along the way).
Skewed isotherms, dry adiabats, and mixing-ratio lines were drifting
outside the frame at the corners — most visibly the warm-side
isotherms exiting the right edge near the top of the diagram. The
existing `clip_polyline/1` only filtered with a ±200 px slack, so
segments crossing the boundary still drew past the frame.
Wrap every grid layer + the T/Td/parcel polylines in a
`<g clip-path="url(#skewt-plot-clip)">`. The clipPath defines a
rectangle from (left, top) to (right, bottom). Frame outline and
axis labels stay outside the clip group so they keep rendering at
the box edge unchanged. The hover-cursor crosshair also lives
outside the clip so its readout box can sit flush against the plot
corner without being trimmed.
mix test: 2,902 / 2,902 + 221 properties green. mix credo --strict
clean.
Two changes that bring /skewt visually and analytically into parity
with the NOAA SPC sounding viewer
(https://www.spc.noaa.gov/exper/soundings/):
NOAA-style line set
-------------------
Updated the SkewtSvg style block to match the public SPC chart:
* Temperature trace ........ pure red (#ff0000) at 2.5 px stroke
* Dewpoint trace ........... pure green (#00aa00) at 2.5 px stroke
* Isotherms ................ dashed teal (#14b8a6)
* Dry adiabats ............. solid orange (#f59e0b)
* Moist (saturated) adiabats dashed sky-blue (#38bdf8)
* Mixing-ratio lines ....... dotted darker green (#16a34a), with
labels at 700 mb showing the g/kg
value, drawn only below 600 mb (matches
the SPC chart's lower-band number row)
* Parcel ascent ............ new dashed grey (#475569) line, drawn
whenever SkewtParams.derive returns a
non-empty parcel_trace
The mixing-ratio set is the SPC standard (0.4 / 0.7 / 1 / 2 / 3 / 5 /
8 / 12 / 16 / 20 g/kg) instead of the prior arbitrary set.
SkewtParams: SPC parcel-theory + indices module
-----------------------------------------------
New `Microwaveprop.Weather.SkewtParams` derives the parameter set that
ships in the SPC `*.txt` companion file:
* LCL pressure / temperature / height (Bolton 1980 eq 22)
* LFC pressure / height
* EL pressure / height
* SBCAPE, SBCIN (J/kg, virtual-temperature integration with
surface parcel)
* 3 km CAPE
* Lifted Index (parcel ↔ environment T at 500 mb)
* Freezing level (T = 0 °C height) and Wet-Bulb Zero
* DCAPE (downdraft CAPE — picks the min-θₑ source level in
400–700 mb, descends moist-adiabatically to surface)
* Layer lapse rates: sfc–3 km, 700–500 mb, 850–500 mb
* `parcel_trace` — pressure/temperature trajectory used by
SkewtSvg to draw the dashed parcel-ascent overlay
Wind-derived indices (SRH, BWD, Bunkers motion, SCP, STP, SHIP) are
deliberately *not* produced and the LiveView surfaces a one-line note
explaining why: HRRR persists wind only at 10 m AGL, so per-pressure-
level shear can't be computed from this data source.
LiveView panel restructured into grouped sections (Surface, Convective
parcel, Levels, Lapse rates, Moisture/stability, Refractivity) so the
layout matches the SPC viewer's order. The Levels rows show pressure
*and* height ("872 mb · 1,140 m") for quick cross-reference between
the diagram axis and the readout.
Tests: 8/8 new SkewtParamsTest cases (LCL Bolton check, saturated-air
edge case, full field-set presence, summer profile produces SBCAPE > 0
and LI < 0, capped profile produces SBCAPE = 0 and LI > 0, freezing
level interpolated correctly, sfc–3 km lapse rate, empty profile no
crash). 25/25 across all skewt-related test files. mix test green
(2,902 / 2,902 + 221 properties), mix credo --strict clean, mix
assets.build clean.
Two follow-ups to the earlier /skewt work:
1. Interactive crosshair (matches the SPC-viewer feel from the
reference screenshot). Mousing over the diagram now drops a
horizontal pressure cursor with a top-left readout box showing the
pressure at the cursor's altitude (mb), the height in m and ft,
and the linearly interpolated temperature and dewpoint at that
level. Two coloured dots track the T and Td traces along the
cursor so the user can read the spread visually.
Mechanics: SkewtSvg.geometry/0 exposes the plot geometry (left,
right, top, bottom, p_bottom, p_top, t_min, t_max, skew) as JSON;
SkewtSvg emits an initially-hidden <g class="skewt-cursor"> layer.
A new Skewt hook in assets/js/app.ts inverts pressure_y to recover
pressure from the cursor's viewBox-Y, walks the (descending-pres-
sorted) profile to interp T/Td/height, and updates the elements.
The hook also handles mouseleave to hide the cursor and uses the
SVG's screen-CTM so the math stays correct under any container
aspect-ratio resize.
2. DB fallback for `available_valid_times`/`load_profile`. When the
on-disk `ProfilesFile` has nothing for the location-and-window
(true on dev, transient on prod between chain runs), SkewtLive
now falls back to `HrrrProfileLookup.{list_valid_times_near,
read_point_near}/3` against the `hrrr_profiles` Postgres table.
The lookup uses the same ±0.07° spatial tolerance and the
`(lat, lon, valid_time)` unique index that `Weather.find_nearest_
hrrr/3` already relies on, so the queries are index-only.
Returned shape matches `ProfilesFile.read_point/3` so SkewtLive
reads from either source transparently. NaiveDateTime → UTC
normalisation is centralised in `ensure_utc/1` so the LiveView
sees `%DateTime{time_zone: "Etc/UTC"}` regardless of which side
produced the value.
Tests: 6/6 new HrrrProfileLookupTest cases (list/limit/tolerance,
read nearest cell, nil for far cells, picks closest among matches),
3/3 SkewtLiveTest still green, 2,894 total Elixir tests + 221
properties green via mix test (one pre-existing flaky test in
admin/contact_edit_live re-runs clean). mix credo --strict clean.
mix assets.build clean.
The /skewt page was empty in prod because Phase 2 of the Rust cutover
moved f01..f18 grid scoring to Rust but left profile-file writing only
on the analysis (f00) path. Result: /data/scores/profiles/ accumulated
one file per chain run instead of nineteen, so SkewtLive only ever saw
the analysis hour — and that hour falls outside the 1 h past cutoff
within an hour of the chain finishing, leaving the page empty most of
the time.
Two surgical fixes:
* `pipeline::run_chain_step` now builds `profile_entries` alongside
`prepared` from the merged grid in a single iter pass, then spawns
`profiles_file::write_atomic` on the blocking pool to overlap the
NFS write with the score-band scoring/write fan-out — same pattern
`run_analysis_step` already uses. `ChainStepStats` gains
`profile_cells_written` so the worker log line is symmetric with
the analysis step's existing field and a future failure mode that
drops the profile would show up as a divergence.
* `SkewtLive.available_valid_times/0` widens the past cutoff from
1 h to 3 h. The chain's analysis hour can be ~70 min old by the
time the next chain finishes (and ~2 h on a missed cycle), so the
1 h cutoff was leaving the page blank during normal transients.
18 h forward window unchanged.
Cosmetic: cargo fmt picked up trivial rewrites in decoder/fetcher/
native_duct/hrrr_point_worker that had drifted; rolled in.
Verification: cargo build --release, cargo clippy --lib --bins -D
warnings, cargo test --lib (134/134), mix test
test/microwaveprop_web/live/skewt_live_test.exs (3/3) all green.
