Move ScalarFile from a dual-writer dual-format setup (ETF on the Elixir
side, materialized via NotifyListener; nothing on the Rust side) to a
single chunked-MessagePack format that both languages produce and
consume.
Elixir side:
- ScalarFile now writes gzipped MessagePack `.mp.gz` per chunk via
Msgpax. Reader decodes with Msgpax + zlib. Atom keys are stripped to
strings on encode and re-atomized via a whitelist on read so callers
see the same shape regardless of which side wrote the bytes.
- DateTime values round-trip through ISO8601 strings.
- New cross-language wire-format test that decodes a hand-rolled
payload mirroring exactly what `rmp_serde::to_vec_named` emits, so a
format regression on either side fails the suite.
Rust side:
- New `prop_grid_rs::weather_scalar_file` module:
- 1:1 port of `Microwaveprop.Weather.WeatherLayers.derive/1` plus
the surface fields `Weather.build_grid_cache_row/4` adds on top
(lapse rate, mid lapse rate, inversion strength + base, 850/700 mb
interpolation, surface RH, surface refractivity, ducting flag).
- `write_atomic` chunks rows into 5°×5° buckets, writes one
`<lat_band>_<lon_band>.mp.gz` per chunk via tmp + rename, drops
pre-existing chunks first to keep writes canonical.
- 5 unit tests covering surface-only rows, upper-air derivation,
write/read round-trip, chunk band boundaries, and the
out-of-physical-range surface-temp filter.
- Pipeline integration: both `run_chain_step` (f01..f18) and the f00
analysis path now build `Vec<ScalarRow>` in the same merged-cell
pass that builds `CellEntry` and `Conditions`, then fire a parallel
`spawn_blocking` write that overlaps with scoring. Awaiting the
scalar future after the score-band writes keeps the failure-surface
symmetric with the existing profile_future.
Result: every new forecast hour lands with the scalar artifact already
on disk, so /weather reads never fall through to ProfilesFile decode +
per-cell SoundingParams + WeatherLayers derivation. The Elixir-side
`NotifyListener.handle_propagation_ready/1` materialization remains as
an idempotent safety net for hours that pre-date this commit.
Mirror ScalarFile's 5°×5° chunk layout inside GridCache. Each ETS entry
is now `%{{lat_band, lon_band} => %{{lat, lon} => row}}` instead of a
flat `%{{lat,lon} => row}`. fetch_bounds/2 walks only the chunks that
intersect the requested viewport, so latest-hour pans become
proportional to visible chunks (~2-4) rather than O(92k cells).
fetch/1, fetch_point/3, put/2 and broadcast_put/2 keep their
pre-existing contracts.
Also closes the open items in docs/weather.md: Phase 3 is now done end
to end. The only remaining future-work item is moving WeatherLayers.derive
itself into the Rust pipeline so the BEAM doesn't pay the one-time
materialization pass per new valid_time — small runtime savings vs.
significant port effort, so left as documented future work.
Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
