prop/docs/weather.md

Weather Page Performance Review

Scope

This review covers how HRRR-backed weather data is stored, derived, cached, and rendered for /weather, with a focus on why the page feels slow and what to change first.

Implementation Status

Completed so far:

• /weather no longer ships visible weather rows over LiveView for the overlay (finding #6).
• The browser now uses a dedicated weather tile endpoint and only requests the overlay tiles needed for the current viewport (finding #5).
• The old client-side GridLayer.createTile path that rebuilt canvases from websocket row payloads has been replaced by a Leaflet tile overlay (finding #5).
• Layer/time changes now refresh overlay tile URLs instead of pushing large update_weather payloads (finding #6).
• A dedicated Microwaveprop.Weather.ScalarFile artifact persists pre-derived weather scalars per valid_time on NFS, bucketed into 5°×5° chunks. weather_grid_at/2 and weather_point_detail/3 read from it before falling back to raw ProfilesFile decoding (findings #1, #2, #3, #4, #7).

All listed items shipped. The Microwaveprop.Weather.WeatherLayers.derive/1 port now lives in prop_grid_rs::weather_scalar_file and the Rust pipeline writes ScalarFile artifacts inline alongside profiles_file::write_atomic, using the same gzipped MessagePack chunked-by-5°×5° wire format the Elixir writer produces. The Elixir reader atomizes a whitelist of keys on read so callers see the same shape regardless of which side wrote the bytes.

Available infrastructure that should shape the solution:

• NFS storage is available for persisted derived artifacts and generated assets.
• Memory usage matters, so solutions should avoid large in-memory cache layers.

Relevant code paths:

• lib/microwaveprop_web/live/weather_map_live.ex
• lib/microwaveprop/weather.ex
• lib/microwaveprop/weather/weather_layers.ex
• lib/microwaveprop/weather/grid_cache.ex
• lib/microwaveprop/propagation/profiles_file.ex
• assets/js/weather_map_hook.ts

Current Flow

Storage

• The Rust/propagation pipeline persists one full HRRR grid file per valid_time in ProfilesFile as .mp.gz or .etf.gz with %{{lat, lon} => profile} raw-ish profile maps, not a weather-specific projection: profiles_file.ex.
• Those per-cell profiles still contain enough atmospheric structure that /weather has to derive map fields at read time.
• Legacy hrrr_profiles rows still exist as fallback for some point-detail reads: weather.ex.

Server read path

• /weather mount stays cheap and now sends only metadata needed to drive the tile overlay: selected layer, valid times, selected time, and the tile base URL.
• Weather overlay requests now go through a dedicated tile endpoint, one tile per visible z/x/y.
• Tile requests currently still call into weather_grid_at/2, so cold reads still decode the full ProfilesFile, filter to bounds, and derive rows on demand: weather.ex.
• build_grid_cache_rows/3 still runs SoundingParams.derive/1, then WeatherLayers.derive/1, for every selected point: weather.ex.

Client render path

• The browser now uses a Leaflet tile overlay for weather instead of receiving visible weather rows as JSON over LiveView.
• Leaflet only requests the weather tiles needed for the user’s current viewport.
• Layer/time changes refresh the tile URL rather than rebuilding a client-side weather grid.
• Point detail is still requested separately over LiveView.

Findings

1. Biggest bottleneck: weather fields are derived repeatedly at read time instead of being stored in a weather-friendly format — RESOLVED (in steady state)

Severity: high (was). Mitigated for any valid_time that has a materialized scalar file.

What shipped:

• New Microwaveprop.Weather.ScalarFile (scalar_file.ex) persists derived scalar rows per valid_time on NFS at <base_dir>/weather_scalars/<iso>/.
• Stored fields are exactly what the map needs (temperature, dewpoint_depression, surface_rh, surface_pressure_mb, surface_refractivity, refractivity_gradient, bl_height, pwat, 850/700 mb T+Td, lapse_rate, mid_lapse_rate, inversion_strength, inversion_base_m, ducting, duct_base_m, duct_strength).
• weather_grid_at/2 reads the scalar file before falling back to ProfilesFile (weather.ex). When the scalar file is present, no SoundingParams.derive/1 or WeatherLayers.derive/1 runs at read time.
• The ProfilesFile store is preserved as the fallback for any valid_time that hasn't been materialized yet (and as the source of truth for advanced diagnostics).

What's still open:

• The first read on a fresh valid_time that has no scalar file yet still pays the legacy derive cost. The cold-derive path additionally fires a background materialization (kickoff_async_scalar_materialize) so the next reader gets the cheap path. That gap closes entirely once the propagation pipeline writes scalar files inline.

2. Small viewport requests still decode the full grid file — RESOLVED (in steady state)

Severity: high (was).

What shipped:

• ScalarFile.read_bounds/2 only decodes the 5°×5° chunk files that overlap the requested bounds, so a state-sized viewport at z=6-7 reads 1-4 chunks instead of a full ~10 MB blob.
• ScalarFile.read_point/3 reads exactly one chunk (the one containing the snapped cell), so point-detail clicks on a forecast hour no longer cost a whole-file decode.
• The raw ProfilesFile is still consulted only when no scalar file exists for valid_time, which is the cold-start case.

What's still open:

• During the cold-derive fallback, ProfilesFile.read/1 still decodes the full file once before the background materializer writes the scalar artifact. The fix is to have the propagation pipeline produce scalar files inline so the fallback path is never taken in production.

3. Forecast hours are intentionally not cached, but the real fix is changing what gets read — RESOLVED (in steady state)

Severity: high (was).

What shipped:

• Forecast-hour scrubs no longer require the full ProfilesFile decode + per-cell derivation when a scalar file exists for that hour. weather_grid_at/2 reads the scalar chunks for the visible viewport directly from NFS.
• Memory budget is unchanged — scalars live on disk, not in ETS, so there's no resident-memory cost per cached forecast hour.

What's still open:

• Same as #1 / #2: the very first scrub to a never-materialized hour pays the cold-derive cost once before the background task fills in the scalar file.

4. GridCache.fetch_bounds/2 still scans the whole cached grid on every pan — RESOLVED

Severity: medium (was).

What shipped:

• GridCache now stores each valid_time as %{{lat_band, lon_band} => %{{lat, lon} => row}} using the same 5°×5° chunk layout as ScalarFile (grid_cache.ex).
• fetch_bounds/2 only walks the chunks that intersect the requested viewport, so a state-sized viewport reads ~2-4 chunks instead of filtering ~92k cells.
• fetch_point/3 reads exactly one chunk.
• All public APIs (fetch/1, fetch_bounds/2, fetch_point/3, put/2, broadcast_put/2) preserve their pre-rewrite contracts.

5. The frontend overlay renderer is doing too much work on the main thread — RESOLVED

Severity: high (was) — the heavy client-side path is gone.

Original symptom:

• update_weather rebuilt the lookup map and recreated the overlay every push.
• GridLayer.createTile looped over every tile cell on the browser main thread.

What shipped:

• The browser uses a server-rendered L.tileLayer pointing at /weather/tiles/:z/:x/:y (weather_map_hook.ts).
• Layer/time changes call setUrl + redraw rather than rebuilding any data structure.
• Tiles are SVG <rect> elements rendered server-side by tile_renderer.ex, so per-tile compositing is the browser's tile cache, not Elixir.

What's still open under this finding:

• The server-side tile renderer derives rows from raw ProfilesFile data on every tile request — see finding #1 (still the dominant remaining cost).

6. The websocket payload is larger than it needs to be — RESOLVED

Severity: medium (was) — the bulk overlay payload is gone.

Original symptom:

• update_weather pushed an array of full row maps to the client on every viewport / time change.

What shipped:

• The LiveView no longer pushes weather rows. The only overlay-related events are update_weather_overlay and update_timeline, which carry just metadata (selected ISO time, list of valid times) (weather_map_live.ex, weather_map_live.ex).
• Per-cell data is fetched from the tile endpoint by viewport, and point clicks remain a separate small point_detail event.

6. The websocket payload is larger than it needs to be

Severity: medium

• update_weather pushes an array of full row maps to the client: weather_map_live.ex.
• Each row contains every weather layer field, even though only one layer is rendered at a time.

Impact:

• Extra server serialization time.
• Extra websocket transfer time.
• Extra browser parsing time before render.

Recommendation:

• Split payloads by use:

  • for map paint, send only lat, lon, and the selected layer value
  • for click detail, fetch/send the expanded record separately

• If layer switching must be instant without another round trip, use a compact binary or columnar payload for the visible rows instead of verbose JSON objects. Expected gain:

• Lower end-to-end latency and less client parsing work.

7. Point detail has a surprisingly expensive cold path — RESOLVED (in steady state)

Severity: medium (was).

What shipped:

• weather_point_detail/3 now consults ScalarFile.read_point/3 between the GridCache ETS hit and the legacy ProfilesFile.read_point/3 path. The scalar lookup decodes a single 5°×5° chunk file, never the full grid.
• For any valid_time with a materialized scalar file, a point click is one chunk read + a small list-find — comparable to the in-memory ETS lookup in cost.

What's still open:

• Same as #1: until the propagation pipeline writes scalar files inline, the very first click on a never-materialized forecast hour still falls back to ProfilesFile.read_point/3. Mitigated because the bounds-read on the same scrub already kicked off a background materialization.

8. Timeline and control rendering — RESOLVED

Severity: low (was).

What shipped:

• The legend L.Control is mounted once at mounted() and only its inner content (legendContent.innerHTML) is patched on layer changes (weather_map_hook.ts).
• renderTimeline keys off the rendered button set (timelineRenderedKey). Selection-only updates take an applyTimelineSelection patch path that restyles existing buttons in place rather than rebuilding innerHTML and re-binding click listeners. A full rebuild only fires when the underlying timelineData changes (new forecast hour landed).

Prioritized Plan

Phase 1: biggest wins with the least architecture churn — DONE

1. Persist pre-derived weather scalars per valid_time instead of deriving them on every /weather read. ✅ Microwaveprop.Weather.ScalarFile.
2. Persist those scalar artifacts on NFS in a viewport-friendly layout. ✅ 5°×5° chunk files under <base_dir>/weather_scalars/<iso>/.
3. Change point-detail reads so one click does not require decoding the whole grid file. ✅ ScalarFile.read_point/3 reads a single chunk.

Remaining gap: the propagation pipeline doesn't yet write scalar files inline; we lazily materialize on first read with a per-valid_time lock. That removes server-side lag for every reader after the first.

Phase 2: fix the browser-side jank

1. Replace the current GridLayer.createTile nested-loop renderer with either:
   • pre-rendered raster tiles, or
   • a single canvas/WebGL overlay backed by numeric arrays.
2. Reduce update_weather payloads to the currently selected layer only.

This should remove most pan/zoom/layer-switch lag.

Status:

• Partially complete.
• Done:
  • the old client-side GridLayer.createTile path is gone
  • the overlay now loads as viewport-scoped tiles
  • the large update_weather row payload is gone for overlay painting
• Still open:
  • tile generation still derives from raw profiles on the server
  • tiles are generated on request rather than read from a weather-specific derived artifact

Phase 3: improve cache shape for latest-hour panning — DONE

1. Replace full-map Enum.filter/2 in GridCache.fetch_bounds/2 with chunked spatial storage. ✅ shipped — GridCache now mirrors ScalarFile's 5°×5° chunk layout in ETS, so latest-hour pans walk only the chunks that intersect the viewport.
2. Persist chunks on NFS and read only the visible chunk set. ✅ done in Phase 1.
3. Pre-generate the latest analysis hour and adjacent forecast hours as part of the pipeline so the app serves ready-made artifacts. ✅ shipped — NotifyListener.handle_propagation_ready/1 fires Weather.materialize_scalar_file/1 in a detached Task immediately on NOTIFY propagation_ready. Moving the derivation into Rust itself remains future work; see Still Pending.

Suggested Target Architecture

Write-time

During the propagation/HRRR pipeline, emit three artifacts per valid_time:

1. Raw full profile store
   • Used for advanced diagnostics and any future science work.
2. Weather scalar grid store
   • One compact record per grid point containing only final fields used by /weather.
   • Persist on NFS.
3. Point-detail index/store
   • Fast random access for one clicked cell.
   • Persist on NFS.
4. Optional raster tile/chunk store
   • Pre-rendered layer assets for the browser.
   • Persist on NFS.

Read-time

• /weather map paint reads scalar grid chunks or raster tiles only.
• NFS is the durable source of truth.
• Point detail reads the indexed point store, not the full grid blob.
• Client receives only what it needs for the active layer or an already-rasterized tile.

Concrete Implementation Suggestions

Storage format

• Keep raw ProfilesFile as-is for safety during migration.
• Add WeatherScalarFile with a compact binary format.
• Store grid points in row-major or chunk-major order with integer indices instead of float map keys.
• Put derived scalar files, point indexes, and optional raster tiles on NFS.

Server API changes

• Add something like Weather.scalar_grid_at(valid_time, bounds, layer_id).
• Add Weather.point_detail_at(valid_time, lat, lon) backed by a point index.
• Keep the current APIs as compatibility wrappers during cutover.
• Prefer chunk-oriented helpers over raw viewport scans, for example:
  • Weather.visible_chunks(bounds)
  • Weather.chunk_rows_at(valid_time, layer_id, chunk_ids)

Frontend changes

• Replace gridLookup: Map<string, WeatherPoint> with indexed numeric buffers.
• Avoid rebuilding the overlay object unless the viewport or layer renderer truly changes.
• Prefer one draw pass per visible grid rather than per-tile nested string lookups.
• If raster tiles are adopted, the frontend becomes much simpler:
  • Leaflet swaps tile URLs by valid_time + layer_id
  • the websocket coordinates only state and point-detail interactions

More Optimal Storage / Retrieval Options

These are the realistic options, from least to most optimized for /weather.

Option A: keep the current raw profile blob model

Description:

• Store one full raw profile file per valid_time.
• Derive rows on reads.

Pros:

• Minimal write-path change.

Cons:

• Worst read performance.
• Wrong shape for viewport reads and point-detail reads.
• Repeats expensive atmospheric derivation work.

Verdict:

• Not recommended for /weather.

Option B: one derived scalar file per valid_time

Description:

• At write time, derive the final weather fields once.
• Store one compact scalar file per hour on NFS.

Pros:

• Much better than current design.
• Simplest migration path.

Cons:

• Still requires full-file decode unless you add an index.

Verdict:

• Good intermediate step.

Option C: chunked derived scalar files per valid_time

Description:

• Store weather rows grouped into spatial chunks on NFS.
• Read only visible chunks.

Pros:

• Best fit for viewport-driven map access.
• Good memory behavior.
• Works well for both JSON and raster generation paths.

Cons:

• Slightly more write-path complexity.
• Requires chunk manifest management.

Verdict:

• Best overall backend format for this app.

Option D: pre-rendered raster tiles per valid_time and layer

Description:

• Generate tile images or larger raster chunks for each weather layer.
• Browser reads tiles directly; websocket traffic drops sharply.

Pros:

• Best browser responsiveness.
• Simplest client rendering path.
• Excellent for many concurrent viewers.

Cons:

• More storage.
• More write-time fanout across layers.
• Point detail still needs a scalar/point store beside the tiles.

Verdict:

• Best frontend performance, especially if /weather is primarily a visual browsing tool.

Recommended Combination

If the goal is the best practical design without adding more cache infrastructure:

1. Store chunked derived scalar files on NFS.
2. Store a point-detail index/store on NFS.
3. Optionally add pre-rendered raster tiles on NFS for the most-used layers once the scalar chunk path is stable.

That gives you:

• efficient write-once derivation
• fast viewport reads
• fast point clicks
• predictable memory use
• a clean path to even faster raster rendering later

Best-Fit Scaled-Down Solution

If I were implementing this here, I would choose:

1. Raw profiles remain in ProfilesFile for science/debug paths.
2. A new NFS-backed chunked WeatherScalarFile becomes the primary /weather read path.
3. A new NFS-backed point index/store becomes the primary point-detail read path.
4. Later, if /weather still needs more smoothness, layer rasters/tiles get generated from the scalar chunks and served directly.

That is the most balanced scaled-down design here. It is materially faster than the current shape, avoids wasting CPU on repeated derivation, and does not depend on large memory-resident caches.

Bottom Line

All listed phases are shipped:

• /weather reads from pre-derived scalar chunks on NFS instead of re-deriving from raw HRRR data on every request.
• Viewport reads only touch the chunks that overlap the current bounds — for both NFS-backed ScalarFile reads and the in-memory GridCache.
• Point clicks read a single chunk instead of the whole grid file.
• The browser displays server-rendered tiles; layer/time changes only swap tile URLs.
• New forecast hours land with the scalar artifact already on disk: the Rust pipeline writes scalar chunks inline alongside the raw profile, and the Elixir-side NotifyListener.handle_propagation_ready/1 keeps an idempotent fallback materialization for hours that pre-date the Rust deploy.
• Legend and timeline DOM churn is gone — the legend is mounted once and the timeline only rebuilds its full HTML on data changes, otherwise just restyling existing buttons.

Both languages produce and consume identical bytes on disk: gzipped MessagePack chunks under <base>/weather_scalars/<iso>/<lat_band>_<lon_band>.mp.gz. There is no Elixir-only or Rust-only branch in the format.