Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
Refit on the current contact corpus via mix recalibrate_scorer (val loss
0.140 vs 0.146 train, initial 0.434). Mostly small adjustments — the
notable shifts are time_of_day −0.0116 (now the smallest factor) and
rain +0.0069 (continues to be the largest single weight).
Sums to 1.0 (within rounding); per-band overrides not affected.
`mix prop.compare` runs both scorers against a recent contact sample
and measures both against achieved contact distance. Each run appends
to priv/calibration/history.jsonl, so trends in alg-ML divergence and
algorithm-distance correlation surface over weeks of running.
When drift exceeds threshold (alg-ML RMSE > 8 score points, or current
correlation > 0.10 below the history median), the task writes a
recommendation pointing at the right remediation:
mix recalibrate_scorer # algorithm drift → refit weights
mix propagation_train # ML drift → retrain on the new algorithm
That is the feedback loop: measure → recommend → recalibrate/retrain →
loop. Operational rather than automatic, so a bad data window can't
silently corrupt the model.
Pure analysis lives in Microwaveprop.Propagation.Calibration with its
own unit tests; the Mix task only handles data loading, file I/O, and
console formatting.
- Unify haversine_km / deg_to_rad: Geo is the canonical home (atan2
form for antipodal stability); Radio, common_volume, rain_scatter,
ionosphere, terrain/srtm, terrain/elevation_client now delegate.
- Drop safe_min/safe_max wrappers in favor of Enum.min/max defaults.
- Move parse_float / parse_int to MicrowavepropWeb.LiveHelpers; eme,
path, and rover LiveViews import from there.
- Extract MicrowavepropWeb.LocationResolver from the eme/path
resolve_source / resolve_location duplicate. Standardize the kind
key and "Invalid grid square" error message.
- Convert Scorer cond chains (humidity, td_depression, sky, wind,
rain, pwat, pressure) to multi-clause guarded defs, matching the
existing classify_time_period style.
- extract_profile_fields uses a named map accumulator instead of a
6-tuple; build_path_conditions guards on %{temps: []} / %{dewpoints: []}.
- Collapse scores_file clamp/3 to max |> min.
- humidity_effect_label lives in BandConfig; map_live and path_live
both use it.
Retrained on 60k month-balanced HRRR profiles through 2026-04-05
(test RMSE 1.7 score points, R² 0.97). Added explain_prediction/4
returning ranked feature contributions via batched finite-difference
attribution, so the UI can show users which weather factors drove
each prediction.
Sweep of remaining swallowed-error sites flagged by an audit pass:
- weather/gefs_client.ex: byte-range download task exits log url + reason
before being surfaced as {:error, _} (matches the HRRR fix).
- live/path_live.ex: terrain-profile load failure, native-duct lookup
failure, and ProfilesFile.read failure now log warnings with path
endpoints / midpoint / valid_time. Previously rendered a degraded
result silently.
- live/rover_live.ex: station-mutation {:error, _} now logs id + reason
instead of dropping into {:noreply, socket}.
- application.ex: the boot-time backfill_missing_home_qth Task.start is
now wrapped in try/rescue. A DB issue at boot would have crashed the
task silently (no Logger handler attached pre-Logger crash format).
GefsFetch silently dropped crashed score-computation tasks via
`{:exit, _reason} -> []`, hiding partial-grid outages from k8s logs.
HRRR range downloads mapped exits to `{:error, reason}` without logging
the URL, so the failing range was lost by the time the caller surfaced
the error.
All three sites now Logger.error with enough context (fh, valid_time,
url) to debug from production logs.
Adds Microwaveprop.Radio.TextSanitizer to remove BOM, zero-width
spaces/joiners, C0 control bytes (except CR/LF/TAB), and DEL, and to
collapse NBSP and other unicode whitespace into a regular space.
CSV: applied to the whole upload at the top of preview/2.
ADIF: applied per-field-value after extraction so the byte-counted
length tags still slice the right substring.
The radar-only RainScatterClassifier sees ducting only when HRRR's
pressure-level M-profile flagged it. Sounding-detected and gradient-
inferred ducts surface in elevation_profile.ducts via the analysis
pipeline — promote those into :tropo_duct so the Mechanism badge
agrees with the narrative.
Click a row's info area to expand a Leaflet mini-map centered on that
pin, defaulting to ESRI World_Imagery satellite (zoom up to 21) with
OSM/Topo toggles in a top-right layer control.
Adds a checkbox in the rover sidebar that constrains the suggested
candidate spots to lat/lon points tagged as :ideal in /rover-locations.
When enabled, Compute.run replaces grid cells with synthetic cells at
each ideal location's exact coordinates, inheriting the propagation
score from the nearest grid cell so atmospheric forecast still factors
in. Path-clearance, terrain, building, and canopy penalties run
unchanged on those snapped candidates.
New globally-shared list of rover-friendly (or off-limits) parking
spots. Anyone can view; logged-in users can add new locations and
edit/delete their own. Each location stores lat/lon, free-text notes,
and a status enum (:ideal | :off_limits).
* Microwaveprop.Rover.Location schema + migration
* Rover.list_locations/0, create_location/2, update_location/3,
delete_location/2 (mutations require User; ownership-scoped)
* /rover-locations LiveView in the public live_session — Add button
is disabled for anonymous visitors, Edit/Delete buttons render
only on the user's own entries
* Tests for context + LiveView covering anonymous read, owner-only
edit/delete, and form submission
Rover ranking now applies a per-cell tree-canopy clutter penalty
(1 dB / 8 m, capped at 4 dB) so cells in dense forests get down-ranked
even when their per-station path clearance survives. PathTerrain
already accounts for trees ON the link path; this surfaces "I'm in a
forest, every direction is foliage."
Canopy.BulkFetch downloads Lang et al. 2020 10 m global canopy-height
3-degree COGs and slices each into 1° × 1° uint8 .canopy tiles via
gdal_translate, the runtime image now ships gdal-bin. Run on prod via:
bin/microwaveprop rpc 'Microwaveprop.Canopy.BulkFetch.run(32.8, -97.0, 200)'
Tile cache lives under /data/canopy/. Already-sliced tiles are skipped.
Adds Microwaveprop.Canopy module that reads 1° × 1° uint8 per-degree
canopy-height tiles (mirrors SRTM .hgt naming/layout, 30 m resolution).
Returns 0 m for areas with no tile on disk so the lookup is safe to
thread through the existing path-clearance pipeline before any data
lands.
Wires canopy into:
- PathTerrain.obstacle_top: per-sample blocker = ground +
max(building_m, canopy_m), so the rover ranks paths through forests
as obstructed even when no buildings sit on the line
- CandidateDetail profile: each sample now carries canopy_m alongside
building_m
- Rover-detail SVG: green "trees" polygon stacked on terrain, under
the red building polygon
- Path calculator elevation chart: green Tree Canopy dataset between
terrain and buildings
Data prep (download Potapov 2019 / Lang 2022 GeoTIFF, slice to per-degree
uint8 tiles, drop in /data/canopy/) is a separate one-shot operation —
without tiles, lookups return 0 and the existing behavior is unchanged.
Path calculator now queries Microsoft Building Footprints for the
tallest structure within 80m of each profile sample and renders a red
"Buildings" dataset on top of the terrain layer in the elevation chart.
Also normalize on-disk grid HRRR cells to include the legacy
:min_refractivity_gradient / :surface_refractivity / :ducting_detected
keys (mapped from :native_min_gradient and :duct_count). Path-calculator
crashed in prod with KeyError when a path's HRRR points came from the
new on-disk grid format instead of the DB profile shape.
Compute.run now accepts a `progress` callback and reports human-readable
labels before each pipeline step (loading grid, looking up elevation,
checking road access, scoring cells, etc.). RoverLive feeds it via
send/2 to itself and renders the latest label as small text to the left
of the Calculate button while it's running.
Each profile sample now carries the tallest building height within 80m
of the path point. The candidate-detail SVG renders these as a red
polygon stacked on top of the terrain so blockage from buildings is
visible alongside ridges. max_obstacle_m now reflects building tops too,
so the clearance label downgrades when buildings sit on the link.
Adds a per-cell building-clutter penalty so the algorithm down-ranks
spots in built environments where buildings on multiple sides
scatter/block signal regardless of which station you're aiming at.
Penalty = max_height_within_75m / 5 dB, capped at 6 dB. Path-clearance
already accounts for buildings ON the link path; this is the
"surrounded by stuff" signal.
Phase 2/3/5 of the building-blockage support:
* Parser streams csv.gz tiles into compact records (centroid, radius,
height) — keeps RAM ~32 B per polygon, drops -1.0-height entries.
* Index buckets records into 0.01° (~1 km) grid cells in a public ETS
table for concurrent reads; max_height_near/3 and records_near/3
scan only the 9 nearest buckets.
* Loader lazily parses any cached quadkey before a Calculate so the
scorer is terrain-only when tiles aren't on disk yet.
* Rover.PathTerrain now treats each path-sample obstacle as
terrain_elev + max_local_building_height, so blocked LOS through
recent construction actually reduces clearance.
* Selecting a candidate pushes a candidate_buildings event with the
buildings near each rover→station path; the hook renders them as
height-coloured circles (yellow <10 m, orange 10-30 m, red >=30 m)
with a tooltip showing height in meters.
Mix tasks don't run in production, so the orchestration lives in
Microwaveprop.Buildings.BulkFetch — callable from an attached iex on
the prod pod via:
iex> Microwaveprop.Buildings.BulkFetch.run(32.8, -97.0, 500)
Default 500 mi radius around DFW yields ~440 candidate quadkeys at
zoom 9; only those present in the MS dataset index (typically a few
hundred over CONUS) get downloaded. Concurrency 4, per-task timeout
3 min, results logged with downloaded/cached/failed/missing counts.
The mix.task is kept as a dev-only thin wrapper.
Phase 1 of building-blockage support: implements quadkey math (Bing/MS
tile encoding), looks up the MS dataset-links index for UnitedStates,
and downloads csv.gz quadkey tiles to /data/buildings (overridable via
:buildings_cache_dir). Index is cached 24h; tiles are written once and
reused. No path-analysis wiring yet — that's the next slice.
