prop/docs/weather.md

Weather Page Performance Review

Scope

This review covers how HRRR-backed weather data is stored, derived, cached, and rendered for /weather, with a focus on why the page feels slow and what to change first.

Implementation Status

Completed so far:

• /weather no longer ships visible weather rows over LiveView for the overlay (finding #6).
• The browser now uses a dedicated weather tile endpoint and only requests the overlay tiles needed for the current viewport (finding #5).
• The old client-side GridLayer.createTile path that rebuilt canvases from websocket row payloads has been replaced by a Leaflet tile overlay (finding #5).
• Layer/time changes now refresh overlay tile URLs instead of pushing large update_weather payloads (finding #6).
• A dedicated Microwaveprop.Weather.ScalarFile artifact persists pre-derived weather scalars per valid_time on NFS, bucketed into 5°×5° chunks. weather_grid_at/2 and weather_point_detail/3 read from it before falling back to raw ProfilesFile decoding (findings #1, #2, #3, #4, #7).

Still pending:

• Have the propagation pipeline write the scalar artifact directly on the Rust side, alongside .mp.gz. Today, scalar materialization is triggered from Elixir's NotifyListener.handle_propagation_ready/1 as soon as the Rust pipeline fires NOTIFY propagation_ready, which closes the gap for steady-state forecast updates but still costs an extra per-valid_time derivation pass on the BEAM.
• Optional: replace GridCache's in-memory %{{lat,lon} => row} map with the same chunk-keyed layout as ScalarFile. Low priority — GridCache only holds the latest valid_time, so its full-map filter is a small fraction of the surrounding work.

Available infrastructure that should shape the solution:

• NFS storage is available for persisted derived artifacts and generated assets.
• Memory usage matters, so solutions should avoid large in-memory cache layers.

Relevant code paths:

• lib/microwaveprop_web/live/weather_map_live.ex
• lib/microwaveprop/weather.ex
• lib/microwaveprop/weather/weather_layers.ex
• lib/microwaveprop/weather/grid_cache.ex
• lib/microwaveprop/propagation/profiles_file.ex
• assets/js/weather_map_hook.ts

Current Flow

Storage

• The Rust/propagation pipeline persists one full HRRR grid file per valid_time in ProfilesFile as .mp.gz or .etf.gz with %{{lat, lon} => profile} raw-ish profile maps, not a weather-specific projection: profiles_file.ex.
• Those per-cell profiles still contain enough atmospheric structure that /weather has to derive map fields at read time.
• Legacy hrrr_profiles rows still exist as fallback for some point-detail reads: weather.ex.

Server read path

• /weather mount stays cheap and now sends only metadata needed to drive the tile overlay: selected layer, valid times, selected time, and the tile base URL.
• Weather overlay requests now go through a dedicated tile endpoint, one tile per visible z/x/y.
• Tile requests currently still call into weather_grid_at/2, so cold reads still decode the full ProfilesFile, filter to bounds, and derive rows on demand: weather.ex.
• build_grid_cache_rows/3 still runs SoundingParams.derive/1, then WeatherLayers.derive/1, for every selected point: weather.ex.

Client render path

• The browser now uses a Leaflet tile overlay for weather instead of receiving visible weather rows as JSON over LiveView.
• Leaflet only requests the weather tiles needed for the user’s current viewport.
• Layer/time changes refresh the tile URL rather than rebuilding a client-side weather grid.
• Point detail is still requested separately over LiveView.

Findings

1. Biggest bottleneck: weather fields are derived repeatedly at read time instead of being stored in a weather-friendly format — RESOLVED (in steady state)

Severity: high (was). Mitigated for any valid_time that has a materialized scalar file.

What shipped:

• New Microwaveprop.Weather.ScalarFile (scalar_file.ex) persists derived scalar rows per valid_time on NFS at <base_dir>/weather_scalars/<iso>/.
• Stored fields are exactly what the map needs (temperature, dewpoint_depression, surface_rh, surface_pressure_mb, surface_refractivity, refractivity_gradient, bl_height, pwat, 850/700 mb T+Td, lapse_rate, mid_lapse_rate, inversion_strength, inversion_base_m, ducting, duct_base_m, duct_strength).
• weather_grid_at/2 reads the scalar file before falling back to ProfilesFile (weather.ex). When the scalar file is present, no SoundingParams.derive/1 or WeatherLayers.derive/1 runs at read time.
• The ProfilesFile store is preserved as the fallback for any valid_time that hasn't been materialized yet (and as the source of truth for advanced diagnostics).

What's still open:

• The first read on a fresh valid_time that has no scalar file yet still pays the legacy derive cost. The cold-derive path additionally fires a background materialization (kickoff_async_scalar_materialize) so the next reader gets the cheap path. That gap closes entirely once the propagation pipeline writes scalar files inline.

2. Small viewport requests still decode the full grid file — RESOLVED (in steady state)

Severity: high (was).

What shipped:

• ScalarFile.read_bounds/2 only decodes the 5°×5° chunk files that overlap the requested bounds, so a state-sized viewport at z=6-7 reads 1-4 chunks instead of a full ~10 MB blob.
• ScalarFile.read_point/3 reads exactly one chunk (the one containing the snapped cell), so point-detail clicks on a forecast hour no longer cost a whole-file decode.
• The raw ProfilesFile is still consulted only when no scalar file exists for valid_time, which is the cold-start case.

What's still open:

• During the cold-derive fallback, ProfilesFile.read/1 still decodes the full file once before the background materializer writes the scalar artifact. The fix is to have the propagation pipeline produce scalar files inline so the fallback path is never taken in production.

3. Forecast hours are intentionally not cached, but the real fix is changing what gets read — RESOLVED (in steady state)

Severity: high (was).

What shipped:

• Forecast-hour scrubs no longer require the full ProfilesFile decode + per-cell derivation when a scalar file exists for that hour. weather_grid_at/2 reads the scalar chunks for the visible viewport directly from NFS.
• Memory budget is unchanged — scalars live on disk, not in ETS, so there's no resident-memory cost per cached forecast hour.

What's still open:

• Same as #1 / #2: the very first scrub to a never-materialized hour pays the cold-derive cost once before the background task fills in the scalar file.

4. GridCache.fetch_bounds/2 still scans the whole cached grid on every pan — partially RESOLVED

Severity: medium (was).

What shipped:

• The expensive code path (weather_grid_at/2 cold reads) now uses chunked spatial reads via ScalarFile. The latest hour also benefits when its grid hasn't been ETS-cached yet.
• The latest-hour GridCache ETS path still does an Enum.filter/2 over the full cached map — we kept it as-is because the in-memory map is small (one valid_time at most) and the filter is the cheap leg compared to the rest of the read path.

What's still open:

• Optionally swap GridCache storage to the same chunk-keyed layout used by ScalarFile so latest-hour pans become O(visible chunks). Low priority since ScalarFile already covers the dominant cost.

5. The frontend overlay renderer is doing too much work on the main thread — RESOLVED

Severity: high (was) — the heavy client-side path is gone.

Original symptom:

• update_weather rebuilt the lookup map and recreated the overlay every push.
• GridLayer.createTile looped over every tile cell on the browser main thread.

What shipped:

• The browser uses a server-rendered L.tileLayer pointing at /weather/tiles/:z/:x/:y (weather_map_hook.ts).
• Layer/time changes call setUrl + redraw rather than rebuilding any data structure.
• Tiles are SVG <rect> elements rendered server-side by tile_renderer.ex, so per-tile compositing is the browser's tile cache, not Elixir.

What's still open under this finding:

• The server-side tile renderer derives rows from raw ProfilesFile data on every tile request — see finding #1 (still the dominant remaining cost).

6. The websocket payload is larger than it needs to be — RESOLVED

Severity: medium (was) — the bulk overlay payload is gone.

Original symptom:

• update_weather pushed an array of full row maps to the client on every viewport / time change.

What shipped:

• The LiveView no longer pushes weather rows. The only overlay-related events are update_weather_overlay and update_timeline, which carry just metadata (selected ISO time, list of valid times) (weather_map_live.ex, weather_map_live.ex).
• Per-cell data is fetched from the tile endpoint by viewport, and point clicks remain a separate small point_detail event.

6. The websocket payload is larger than it needs to be

Severity: medium

• update_weather pushes an array of full row maps to the client: weather_map_live.ex.
• Each row contains every weather layer field, even though only one layer is rendered at a time.

Impact:

• Extra server serialization time.
• Extra websocket transfer time.
• Extra browser parsing time before render.

Recommendation:

• Split payloads by use:

  • for map paint, send only lat, lon, and the selected layer value
  • for click detail, fetch/send the expanded record separately

• If layer switching must be instant without another round trip, use a compact binary or columnar payload for the visible rows instead of verbose JSON objects. Expected gain:

• Lower end-to-end latency and less client parsing work.

7. Point detail has a surprisingly expensive cold path — RESOLVED (in steady state)

Severity: medium (was).

What shipped:

• weather_point_detail/3 now consults ScalarFile.read_point/3 between the GridCache ETS hit and the legacy ProfilesFile.read_point/3 path. The scalar lookup decodes a single 5°×5° chunk file, never the full grid.
• For any valid_time with a materialized scalar file, a point click is one chunk read + a small list-find — comparable to the in-memory ETS lookup in cost.

What's still open:

• Same as #1: until the propagation pipeline writes scalar files inline, the very first click on a never-materialized forecast hour still falls back to ProfilesFile.read_point/3. Mitigated because the bounds-read on the same scrub already kicked off a background materialization.

8. Timeline and control rendering — RESOLVED

Severity: low (was).

What shipped:

• The legend L.Control is mounted once at mounted() and only its inner content (legendContent.innerHTML) is patched on layer changes (weather_map_hook.ts).
• renderTimeline keys off the rendered button set (timelineRenderedKey). Selection-only updates take an applyTimelineSelection patch path that restyles existing buttons in place rather than rebuilding innerHTML and re-binding click listeners. A full rebuild only fires when the underlying timelineData changes (new forecast hour landed).

Prioritized Plan

Phase 1: biggest wins with the least architecture churn — DONE

1. Persist pre-derived weather scalars per valid_time instead of deriving them on every /weather read. ✅ Microwaveprop.Weather.ScalarFile.
2. Persist those scalar artifacts on NFS in a viewport-friendly layout. ✅ 5°×5° chunk files under <base_dir>/weather_scalars/<iso>/.
3. Change point-detail reads so one click does not require decoding the whole grid file. ✅ ScalarFile.read_point/3 reads a single chunk.

Remaining gap: the propagation pipeline doesn't yet write scalar files inline; we lazily materialize on first read with a per-valid_time lock. That removes server-side lag for every reader after the first.

Phase 2: fix the browser-side jank

1. Replace the current GridLayer.createTile nested-loop renderer with either:
   • pre-rendered raster tiles, or
   • a single canvas/WebGL overlay backed by numeric arrays.
2. Reduce update_weather payloads to the currently selected layer only.

This should remove most pan/zoom/layer-switch lag.

Status:

• Partially complete.
• Done:
  • the old client-side GridLayer.createTile path is gone
  • the overlay now loads as viewport-scoped tiles
  • the large update_weather row payload is gone for overlay painting
• Still open:
  • tile generation still derives from raw profiles on the server
  • tiles are generated on request rather than read from a weather-specific derived artifact

Phase 3: improve cache shape for latest-hour panning

1. Replace full-map Enum.filter/2 in GridCache.fetch_bounds/2 with chunked spatial storage. (Optional now — ScalarFile covers the dominant viewport-read cost. GridCache is a thin in-memory hot path with one entry, so the filter is cheap relative to the surrounding work.)
2. Persist chunks on NFS and read only the visible chunk set. ✅ done in Phase 1.
3. Pre-generate the latest analysis hour and adjacent forecast hours as part of the pipeline so the app serves ready-made artifacts. Pending — the propagation pipeline (Rust side) should write the scalar artifact alongside .mp.gz. Until then, the first read on each new hour does a one-time materialization on the BEAM.

Suggested Target Architecture

Write-time

During the propagation/HRRR pipeline, emit three artifacts per valid_time:

1. Raw full profile store
   • Used for advanced diagnostics and any future science work.
2. Weather scalar grid store
   • One compact record per grid point containing only final fields used by /weather.
   • Persist on NFS.
3. Point-detail index/store
   • Fast random access for one clicked cell.
   • Persist on NFS.
4. Optional raster tile/chunk store
   • Pre-rendered layer assets for the browser.
   • Persist on NFS.

Read-time

• /weather map paint reads scalar grid chunks or raster tiles only.
• NFS is the durable source of truth.
• Point detail reads the indexed point store, not the full grid blob.
• Client receives only what it needs for the active layer or an already-rasterized tile.

Concrete Implementation Suggestions

Storage format

• Keep raw ProfilesFile as-is for safety during migration.
• Add WeatherScalarFile with a compact binary format.
• Store grid points in row-major or chunk-major order with integer indices instead of float map keys.
• Put derived scalar files, point indexes, and optional raster tiles on NFS.

Server API changes

• Add something like Weather.scalar_grid_at(valid_time, bounds, layer_id).
• Add Weather.point_detail_at(valid_time, lat, lon) backed by a point index.
• Keep the current APIs as compatibility wrappers during cutover.
• Prefer chunk-oriented helpers over raw viewport scans, for example:
  • Weather.visible_chunks(bounds)
  • Weather.chunk_rows_at(valid_time, layer_id, chunk_ids)

Frontend changes

• Replace gridLookup: Map<string, WeatherPoint> with indexed numeric buffers.
• Avoid rebuilding the overlay object unless the viewport or layer renderer truly changes.
• Prefer one draw pass per visible grid rather than per-tile nested string lookups.
• If raster tiles are adopted, the frontend becomes much simpler:
  • Leaflet swaps tile URLs by valid_time + layer_id
  • the websocket coordinates only state and point-detail interactions

More Optimal Storage / Retrieval Options

These are the realistic options, from least to most optimized for /weather.

Option A: keep the current raw profile blob model

Description:

• Store one full raw profile file per valid_time.
• Derive rows on reads.

Pros:

• Minimal write-path change.

Cons:

• Worst read performance.
• Wrong shape for viewport reads and point-detail reads.
• Repeats expensive atmospheric derivation work.

Verdict:

• Not recommended for /weather.

Option B: one derived scalar file per valid_time

Description:

• At write time, derive the final weather fields once.
• Store one compact scalar file per hour on NFS.

Pros:

• Much better than current design.
• Simplest migration path.

Cons:

• Still requires full-file decode unless you add an index.

Verdict:

• Good intermediate step.

Option C: chunked derived scalar files per valid_time

Description:

• Store weather rows grouped into spatial chunks on NFS.
• Read only visible chunks.

Pros:

• Best fit for viewport-driven map access.
• Good memory behavior.
• Works well for both JSON and raster generation paths.

Cons:

• Slightly more write-path complexity.
• Requires chunk manifest management.

Verdict:

• Best overall backend format for this app.

Option D: pre-rendered raster tiles per valid_time and layer

Description:

• Generate tile images or larger raster chunks for each weather layer.
• Browser reads tiles directly; websocket traffic drops sharply.

Pros:

• Best browser responsiveness.
• Simplest client rendering path.
• Excellent for many concurrent viewers.

Cons:

• More storage.
• More write-time fanout across layers.
• Point detail still needs a scalar/point store beside the tiles.

Verdict:

• Best frontend performance, especially if /weather is primarily a visual browsing tool.

Recommended Combination

If the goal is the best practical design without adding more cache infrastructure:

1. Store chunked derived scalar files on NFS.
2. Store a point-detail index/store on NFS.
3. Optionally add pre-rendered raster tiles on NFS for the most-used layers once the scalar chunk path is stable.

That gives you:

• efficient write-once derivation
• fast viewport reads
• fast point clicks
• predictable memory use
• a clean path to even faster raster rendering later

Best-Fit Scaled-Down Solution

If I were implementing this here, I would choose:

1. Raw profiles remain in ProfilesFile for science/debug paths.
2. A new NFS-backed chunked WeatherScalarFile becomes the primary /weather read path.
3. A new NFS-backed point index/store becomes the primary point-detail read path.
4. Later, if /weather still needs more smoothness, layer rasters/tiles get generated from the scalar chunks and served directly.

That is the most balanced scaled-down design here. It is materially faster than the current shape, avoids wasting CPU on repeated derivation, and does not depend on large memory-resident caches.

Bottom Line

The structural fixes are now in place:

• /weather reads from pre-derived scalar chunks on NFS instead of re-deriving from raw HRRR data on every request.
• Viewport reads only touch the chunks that overlap the current bounds.
• Point clicks read a single chunk instead of the whole grid file.
• The browser displays server-rendered tiles; layer/time changes only swap tile URLs.
• New forecast hours are pre-warmed: NotifyListener.handle_propagation_ready/1 fires Weather.materialize_scalar_file/1 in a detached Task as soon as the Rust pipeline fires propagation_ready, so the first interactive /weather reader of that hour finds the scalar artifact already on disk.
• Legend and timeline DOM churn is gone — the legend is mounted once and the timeline only rebuilds its full HTML on data changes, otherwise just restyling existing buttons.

Remaining work is one optional optimization (move scalar-file production into the Rust pipeline so we save the one-time BEAM derive on each new forecast hour) and a low-priority GridCache chunk rewrite.