The ingress is cloudflared (Cloudflare Tunnel), which keeps an HTTP/2
connection pool to upstream Service IPs and reuses connections
aggressively. After a pod is removed from Service endpoints,
cloudflared can still hold in-flight HTTP/2 streams to the dying pod
for a noticeable window — 10 s wasn't enough, deploys still bounced
502 on LiveView reconnect.
Bump preStop sleep 10s → 25s and terminationGracePeriodSeconds 40s →
60s so cloudflared has time to turn over its upstream connections
before Phoenix begins draining.
Deploys were causing 502 Bad Gateway on LiveView reconnect. The kubelet
sent SIGTERM to the old pod and Phoenix immediately stopped accepting
connections, but the Service endpoint controller and ingress hadn't
yet propagated the pod's removal from endpoints — the browser's
reconnect raced the endpoint update and bounced 502 against the dead
pod, then gave up.
Add a 10s preStop sleep on the container and bump
terminationGracePeriodSeconds to 40 so the load balancer has time to
drain this pod before Phoenix begins shutting down.
The HRRR scalar cache can hold a previous run's forecast hours past
their useful life — the timeline was offering "current conditions"
points that were 2-3 hours old, which is more misleading than helpful
on a map labeled "Now".
Filter Weather.available_weather_valid_times() through a `now - 1h`
cutoff in mount, :weather_updated, and :propagation_pipeline_progress
so the timeline only ever shows the most recent analysis hour plus
forecast hours.
maxUnavailable: 1 was killing the only pod before the new one was
ready, taking the site down for the entire image pull + boot window
on every deploy. Switch to maxSurge: 1 / maxUnavailable: 0 so K8s
spins up the replacement and waits for its readinessProbe to pass
before tearing down the old pod.
The client moved to the binary cell-pack and canvas overlay last
commit, so the per-tile SVG endpoint, the TileRenderer module, and the
JSON shape of /weather/cells are all dead code. Remove them.
/weather/cells now always returns the binary WCEL pack (the ?format=bin
toggle is gone — there's only one format). The data-weather-tile-url
attribute and the weather_tile_url assign are dropped from the LiveView.
Also fix a latent bug surfaced once tests covered the default-layers
path: nil row values were writing the atom :nan into the binary segment
at runtime. Replaced with the explicit f32 quiet-NaN bit pattern so
Number.isNaN flags them correctly on the JS side.
Three changes that shrink first-paint and make layer toggles instant:
1. Server: /weather/cells now supports ?format=bin and ?layers=a,b,c. The
binary "WCEL" pack is ~3× smaller than JSON (Float32 columns, no key
overhead) and parses with DataView + typed-array views — essentially
memcpy on the client.
2. Client: replaced JSON parse + per-cell SVG <rect> with a custom
Leaflet canvas layer. The pack is columnar (lats/lons + Float32Array
per layer) and each draw is one fillRect per cell against a
precomputed 256-step color LUT. At low zoom (CONUS, ~80k cells) SVG
paint cost dominated; canvas keeps it under one frame.
3. Lazy prefetch: the initial fetch only requests the *current* layer
(~1/19 the bytes). Right after paint, a single background fetch
pulls every other layer into the same pack. Layer switches that land
after the prefetch finishes are pure recolor — no network, no
server CPU.
The /weather/tiles endpoint stays for backward compat. The JSON path of
/weather/cells stays too — the binary path is opt-in via ?format=bin.
Previously every layer toggle triggered 21 fresh tile requests with the
new layer in the URL — ~21 server-side SVG renders, plus the round-trip
overhead. Even with the GridCache fix the server still spent real CPU
generating thousands of <rect>s per tile, and the user felt it.
Add /weather/cells: a single JSON endpoint returning every layer field
for every cell in the requested viewport. The hook fetches once on
mount, time change, or pan/zoom, caches the cells in memory, and paints
them via L.svgOverlay. Layer switches now re-color the same cached
cells locally — zero network, zero server CPU.
The single SVG uses lat/lon coords with preserveAspectRatio="none";
Leaflet stretches it linearly to the bbox. There is mild equirect→
mercator distortion at low zoom over CONUS but it's imperceptible at
typical use (z6+) and the layer-toggle UX win is large.
Tile endpoint kept for back-compat. svgOverlay opacity matches the
prior tile renderer's 0.55 fill-opacity.
Each /weather/tiles request was running the full ScalarFile decode path
(File.read → gunzip → Msgpax.unpack → normalize) for any non-analysis
valid_time, because GridCache deliberately skipped forecast hours to keep
memory low. With 21 simultaneous viewport tiles all hitting the same few
chunk files, each tile took 1.5-3.2s of contended IO + CPU.
On a GridCache miss when a ScalarFile exists, hydrate GridCache from the
file once (deduped via the existing claim_fill primitive) and serve all
subsequent reads from ETS. Bound memory with prune_keep_latest/1 — keep
the 24 most-recently-touched valid_times, which covers a full HRRR run
(analysis + 18 forecasts) plus a few stragglers from the previous run.
In the steady state, the first viewport read for a given forecast hour
takes one full ScalarFile decode (~50-100 ms for 92k cells); every
subsequent tile, point lookup, and viewport read for the same hour
serves from ETS in <1 ms.
renderLayer was called during mounted() before selectedTime was seeded
from the dataset, so buildWeatherTileUrl returned null and no tile
layer was added. The overlay only appeared when the user clicked a
forecast hour and update_weather_overlay fired. Seed selectedTime
(and timelineData) from data attributes before the first renderLayer.
Also pulls db_connection 2.9.0 → 2.10.0 and spitfire 0.3.10 → 0.3.11 transitively.
oban_pro 1.6.14 adds suspended state support and conditional __opts__ @impl
to align with Oban v2.21+'s new Worker callback.
Set minReplicas == maxReplicas == 1 on the `prop` HPA so the cluster
runs exactly one app pod. HPA still owns the replica count (Flux
ignores spec.replicas on the Deployment), so this is the right knob —
no need to touch deployment.yaml. Restore autoscaling by raising both
bounds together.
Pre-existing CI failure (`tests/scorer_golden.rs`) — the Rust scorer
was still on the pre-recalibration weight vector, while the Elixir
side picked up new gradient-descent-fit weights in f98ee153.
- Update `DEFAULT_WEIGHTS` in band_config.rs to match the new Elixir
values (humidity 0.1262, time_of_day 0.0380, td_depression 0.1010,
refractivity 0.0986, sky 0.0841, season 0.1134, wind 0.0841,
rain 0.1431, pressure 0.0967, pwat 0.1147).
- Loosen the sum-to-one tolerance in `default_weights_sum_to_one`
from 1e-9 to 1e-3 to match Elixir's `assert_in_delta total, 1.0,
0.001` — gradient-descent fits land near 1.0 but not exact.
- Regenerate `tests/scores.bincode` from `mix rust.golden` so the
golden fixture reflects the recalibrated scorer outputs.
`cargo clippy --all-targets -- -D warnings` clean.
139 Rust unit tests + golden fixture pass.
3072 Elixir tests pass.
Clippy's `neg_cmp_op_on_partial_ord` lint fails on `-D warnings`,
which CI enforces. Using `<=` directly is clearer and equivalent for
the f64 inputs we pass (both are caller-supplied target pressures
that come from the level-key constants — neither is ever NaN).
Move ScalarFile from a dual-writer dual-format setup (ETF on the Elixir
side, materialized via NotifyListener; nothing on the Rust side) to a
single chunked-MessagePack format that both languages produce and
consume.
Elixir side:
- ScalarFile now writes gzipped MessagePack `.mp.gz` per chunk via
Msgpax. Reader decodes with Msgpax + zlib. Atom keys are stripped to
strings on encode and re-atomized via a whitelist on read so callers
see the same shape regardless of which side wrote the bytes.
- DateTime values round-trip through ISO8601 strings.
- New cross-language wire-format test that decodes a hand-rolled
payload mirroring exactly what `rmp_serde::to_vec_named` emits, so a
format regression on either side fails the suite.
Rust side:
- New `prop_grid_rs::weather_scalar_file` module:
- 1:1 port of `Microwaveprop.Weather.WeatherLayers.derive/1` plus
the surface fields `Weather.build_grid_cache_row/4` adds on top
(lapse rate, mid lapse rate, inversion strength + base, 850/700 mb
interpolation, surface RH, surface refractivity, ducting flag).
- `write_atomic` chunks rows into 5°×5° buckets, writes one
`<lat_band>_<lon_band>.mp.gz` per chunk via tmp + rename, drops
pre-existing chunks first to keep writes canonical.
- 5 unit tests covering surface-only rows, upper-air derivation,
write/read round-trip, chunk band boundaries, and the
out-of-physical-range surface-temp filter.
- Pipeline integration: both `run_chain_step` (f01..f18) and the f00
analysis path now build `Vec<ScalarRow>` in the same merged-cell
pass that builds `CellEntry` and `Conditions`, then fire a parallel
`spawn_blocking` write that overlaps with scoring. Awaiting the
scalar future after the score-band writes keeps the failure-surface
symmetric with the existing profile_future.
Result: every new forecast hour lands with the scalar artifact already
on disk, so /weather reads never fall through to ProfilesFile decode +
per-cell SoundingParams + WeatherLayers derivation. The Elixir-side
`NotifyListener.handle_propagation_ready/1` materialization remains as
an idempotent safety net for hours that pre-date this commit.
Mirror ScalarFile's 5°×5° chunk layout inside GridCache. Each ETS entry
is now `%{{lat_band, lon_band} => %{{lat, lon} => row}}` instead of a
flat `%{{lat,lon} => row}`. fetch_bounds/2 walks only the chunks that
intersect the requested viewport, so latest-hour pans become
proportional to visible chunks (~2-4) rather than O(92k cells).
fetch/1, fetch_point/3, put/2 and broadcast_put/2 keep their
pre-existing contracts.
Also closes the open items in docs/weather.md: Phase 3 is now done end
to end. The only remaining future-work item is moving WeatherLayers.derive
itself into the Rust pipeline so the BEAM doesn't pay the one-time
materialization pass per new valid_time — small runtime savings vs.
significant port effort, so left as documented future work.
Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
Refit on the current contact corpus via mix recalibrate_scorer (val loss
0.140 vs 0.146 train, initial 0.434). Mostly small adjustments — the
notable shifts are time_of_day −0.0116 (now the smallest factor) and
rain +0.0069 (continues to be the largest single weight).
Sums to 1.0 (within rounding); per-band overrides not affected.
`mix prop.compare` runs both scorers against a recent contact sample
and measures both against achieved contact distance. Each run appends
to priv/calibration/history.jsonl, so trends in alg-ML divergence and
algorithm-distance correlation surface over weeks of running.
When drift exceeds threshold (alg-ML RMSE > 8 score points, or current
correlation > 0.10 below the history median), the task writes a
recommendation pointing at the right remediation:
mix recalibrate_scorer # algorithm drift → refit weights
mix propagation_train # ML drift → retrain on the new algorithm
That is the feedback loop: measure → recommend → recalibrate/retrain →
loop. Operational rather than automatic, so a bad data window can't
silently corrupt the model.
Pure analysis lives in Microwaveprop.Propagation.Calibration with its
own unit tests; the Mix task only handles data loading, file I/O, and
console formatting.
- Unify haversine_km / deg_to_rad: Geo is the canonical home (atan2
form for antipodal stability); Radio, common_volume, rain_scatter,
ionosphere, terrain/srtm, terrain/elevation_client now delegate.
- Drop safe_min/safe_max wrappers in favor of Enum.min/max defaults.
- Move parse_float / parse_int to MicrowavepropWeb.LiveHelpers; eme,
path, and rover LiveViews import from there.
- Extract MicrowavepropWeb.LocationResolver from the eme/path
resolve_source / resolve_location duplicate. Standardize the kind
key and "Invalid grid square" error message.
- Convert Scorer cond chains (humidity, td_depression, sky, wind,
rain, pwat, pressure) to multi-clause guarded defs, matching the
existing classify_time_period style.
- extract_profile_fields uses a named map accumulator instead of a
6-tuple; build_path_conditions guards on %{temps: []} / %{dewpoints: []}.
- Collapse scores_file clamp/3 to max |> min.
- humidity_effect_label lives in BandConfig; map_live and path_live
both use it.
Retrained on 60k month-balanced HRRR profiles through 2026-04-05
(test RMSE 1.7 score points, R² 0.97). Added explain_prediction/4
returning ranked feature contributions via batched finite-difference
attribution, so the UI can show users which weather factors drove
each prediction.
Sweep of remaining swallowed-error sites flagged by an audit pass:
- weather/gefs_client.ex: byte-range download task exits log url + reason
before being surfaced as {:error, _} (matches the HRRR fix).
- live/path_live.ex: terrain-profile load failure, native-duct lookup
failure, and ProfilesFile.read failure now log warnings with path
endpoints / midpoint / valid_time. Previously rendered a degraded
result silently.
- live/rover_live.ex: station-mutation {:error, _} now logs id + reason
instead of dropping into {:noreply, socket}.
- application.ex: the boot-time backfill_missing_home_qth Task.start is
now wrapped in try/rescue. A DB issue at boot would have crashed the
task silently (no Logger handler attached pre-Logger crash format).
GefsFetch silently dropped crashed score-computation tasks via
`{:exit, _reason} -> []`, hiding partial-grid outages from k8s logs.
HRRR range downloads mapped exits to `{:error, reason}` without logging
the URL, so the failing range was lost by the time the caller surfaced
the error.
All three sites now Logger.error with enough context (fh, valid_time,
url) to debug from production logs.
Adds Microwaveprop.Radio.TextSanitizer to remove BOM, zero-width
spaces/joiners, C0 control bytes (except CR/LF/TAB), and DEL, and to
collapse NBSP and other unicode whitespace into a regular space.
CSV: applied to the whole upload at the top of preview/2.
ADIF: applied per-field-value after extraction so the byte-counted
length tags still slice the right substring.
The radar-only RainScatterClassifier sees ducting only when HRRR's
pressure-level M-profile flagged it. Sounding-detected and gradient-
inferred ducts surface in elevation_profile.ducts via the analysis
pipeline — promote those into :tropo_duct so the Mechanism badge
agrees with the narrative.
Click a row's info area to expand a Leaflet mini-map centered on that
pin, defaulting to ESRI World_Imagery satellite (zoom up to 21) with
OSM/Topo toggles in a top-right layer control.
Adds a checkbox in the rover sidebar that constrains the suggested
candidate spots to lat/lon points tagged as :ideal in /rover-locations.
When enabled, Compute.run replaces grid cells with synthetic cells at
each ideal location's exact coordinates, inheriting the propagation
score from the nearest grid cell so atmospheric forecast still factors
in. Path-clearance, terrain, building, and canopy penalties run
unchanged on those snapped candidates.
New globally-shared list of rover-friendly (or off-limits) parking
spots. Anyone can view; logged-in users can add new locations and
edit/delete their own. Each location stores lat/lon, free-text notes,
and a status enum (:ideal | :off_limits).
* Microwaveprop.Rover.Location schema + migration
* Rover.list_locations/0, create_location/2, update_location/3,
delete_location/2 (mutations require User; ownership-scoped)
* /rover-locations LiveView in the public live_session — Add button
is disabled for anonymous visitors, Edit/Delete buttons render
only on the user's own entries
* Tests for context + LiveView covering anonymous read, owner-only
edit/delete, and form submission
Rover ranking now applies a per-cell tree-canopy clutter penalty
(1 dB / 8 m, capped at 4 dB) so cells in dense forests get down-ranked
even when their per-station path clearance survives. PathTerrain
already accounts for trees ON the link path; this surfaces "I'm in a
forest, every direction is foliage."
Canopy.BulkFetch downloads Lang et al. 2020 10 m global canopy-height
3-degree COGs and slices each into 1° × 1° uint8 .canopy tiles via
gdal_translate, the runtime image now ships gdal-bin. Run on prod via:
bin/microwaveprop rpc 'Microwaveprop.Canopy.BulkFetch.run(32.8, -97.0, 200)'
Tile cache lives under /data/canopy/. Already-sliced tiles are skipped.
Adds Microwaveprop.Canopy module that reads 1° × 1° uint8 per-degree
canopy-height tiles (mirrors SRTM .hgt naming/layout, 30 m resolution).
Returns 0 m for areas with no tile on disk so the lookup is safe to
thread through the existing path-clearance pipeline before any data
lands.
Wires canopy into:
- PathTerrain.obstacle_top: per-sample blocker = ground +
max(building_m, canopy_m), so the rover ranks paths through forests
as obstructed even when no buildings sit on the line
- CandidateDetail profile: each sample now carries canopy_m alongside
building_m
- Rover-detail SVG: green "trees" polygon stacked on terrain, under
the red building polygon
- Path calculator elevation chart: green Tree Canopy dataset between
terrain and buildings
Data prep (download Potapov 2019 / Lang 2022 GeoTIFF, slice to per-degree
uint8 tiles, drop in /data/canopy/) is a separate one-shot operation —
without tiles, lookups return 0 and the existing behavior is unchanged.
Path calculator now queries Microsoft Building Footprints for the
tallest structure within 80m of each profile sample and renders a red
"Buildings" dataset on top of the terrain layer in the elevation chart.
Also normalize on-disk grid HRRR cells to include the legacy
:min_refractivity_gradient / :surface_refractivity / :ducting_detected
keys (mapped from :native_min_gradient and :duct_count). Path-calculator
crashed in prod with KeyError when a path's HRRR points came from the
new on-disk grid format instead of the DB profile shape.
Compute.run now accepts a `progress` callback and reports human-readable
labels before each pipeline step (loading grid, looking up elevation,
checking road access, scoring cells, etc.). RoverLive feeds it via
send/2 to itself and renders the latest label as small text to the left
of the Calculate button while it's running.
