# Weather Page Performance Review

## Scope

This review covers how HRRR-backed weather data is stored, derived, cached, and rendered for `/weather`, with a focus on why the page feels slow and what to change first.

## Implementation Status

Completed so far:

- `/weather` no longer ships visible weather rows over LiveView for the overlay (finding #6).
- The browser now uses a dedicated weather tile endpoint and only requests the overlay tiles needed for the current viewport (finding #5).
- The old client-side `GridLayer.createTile` path that rebuilt canvases from websocket row payloads has been replaced by a Leaflet tile overlay (finding #5).
- Layer/time changes now refresh overlay tile URLs instead of pushing large `update_weather` payloads (finding #6).
- A dedicated `Microwaveprop.Weather.ScalarFile` artifact persists pre-derived weather scalars per `valid_time` on NFS, bucketed into 5°×5° chunks. `weather_grid_at/2` and `weather_point_detail/3` read from it before falling back to raw `ProfilesFile` decoding (findings #1, #2, #3, #4, #7).

All listed items shipped. The `Microwaveprop.Weather.WeatherLayers.derive/1` port now lives in [`prop_grid_rs::weather_scalar_file`](/Users/graham/dev/ntms/microwaveprop/rust/prop_grid_rs/src/weather_scalar_file.rs:1) and the Rust pipeline writes `ScalarFile` artifacts inline alongside `profiles_file::write_atomic`, using the same gzipped MessagePack chunked-by-5°×5° wire format the Elixir writer produces. The Elixir reader atomizes a whitelist of keys on read so callers see the same shape regardless of which side wrote the bytes.

Available infrastructure that should shape the solution:

- NFS storage is available for persisted derived artifacts and generated assets.
- Memory usage matters, so solutions should avoid large in-memory cache layers.

Relevant code paths:

- [`lib/microwaveprop_web/live/weather_map_live.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop_web/live/weather_map_live.ex:167)
- [`lib/microwaveprop/weather.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather.ex:859)
- [`lib/microwaveprop/weather/weather_layers.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather/weather_layers.ex:17)
- [`lib/microwaveprop/weather/grid_cache.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather/grid_cache.ex:1)
- [`lib/microwaveprop/propagation/profiles_file.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/propagation/profiles_file.ex:1)
- [`assets/js/weather_map_hook.ts`](/Users/graham/dev/ntms/microwaveprop/assets/js/weather_map_hook.ts:426)

## Current Flow

### Storage

- The Rust/propagation pipeline persists one full HRRR grid file per `valid_time` in `ProfilesFile` as `.mp.gz` or `.etf.gz` with `%{{lat, lon} => profile}` raw-ish profile maps, not a weather-specific projection: [`profiles_file.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/propagation/profiles_file.ex:120).
- Those per-cell profiles still contain enough atmospheric structure that `/weather` has to derive map fields at read time.
- Legacy `hrrr_profiles` rows still exist as fallback for some point-detail reads: [`weather.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather.ex:1156).

### Server read path

- `/weather` mount stays cheap and now sends only metadata needed to drive the tile overlay: selected layer, valid times, selected time, and the tile base URL.
- Weather overlay requests now go through a dedicated tile endpoint, one tile per visible `z/x/y`.
- Tile requests currently still call into `weather_grid_at/2`, so cold reads still decode the full `ProfilesFile`, filter to bounds, and derive rows on demand: [`weather.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather.ex:919).
- `build_grid_cache_rows/3` still runs `SoundingParams.derive/1`, then `WeatherLayers.derive/1`, for every selected point: [`weather.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather.ex:1043).

### Client render path

- The browser now uses a Leaflet tile overlay for weather instead of receiving visible weather rows as JSON over LiveView.
- Leaflet only requests the weather tiles needed for the user’s current viewport.
- Layer/time changes refresh the tile URL rather than rebuilding a client-side weather grid.
- Point detail is still requested separately over LiveView.

## Findings

### 1. Biggest bottleneck: weather fields are derived repeatedly at read time instead of being stored in a weather-friendly format — RESOLVED (in steady state)

Severity: high (was). Mitigated for any `valid_time` that has a materialized scalar file.

What shipped:

- New `Microwaveprop.Weather.ScalarFile` ([`scalar_file.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather/scalar_file.ex:1)) persists derived scalar rows per `valid_time` on NFS at `<base_dir>/weather_scalars/<iso>/`.
- Stored fields are exactly what the map needs (`temperature`, `dewpoint_depression`, `surface_rh`, `surface_pressure_mb`, `surface_refractivity`, `refractivity_gradient`, `bl_height`, `pwat`, 850/700 mb T+Td, `lapse_rate`, `mid_lapse_rate`, `inversion_strength`, `inversion_base_m`, `ducting`, `duct_base_m`, `duct_strength`).
- `weather_grid_at/2` reads the scalar file before falling back to `ProfilesFile` ([`weather.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather.ex:925)). When the scalar file is present, no `SoundingParams.derive/1` or `WeatherLayers.derive/1` runs at read time.
- The `ProfilesFile` store is preserved as the fallback for any `valid_time` that hasn't been materialized yet (and as the source of truth for advanced diagnostics).

What's still open:

- The first read on a fresh `valid_time` that has no scalar file yet still pays the legacy derive cost. The cold-derive path additionally fires a background materialization (`kickoff_async_scalar_materialize`) so the *next* reader gets the cheap path. That gap closes entirely once the propagation pipeline writes scalar files inline.

### 2. Small viewport requests still decode the full grid file — RESOLVED (in steady state)

Severity: high (was).

What shipped:

- `ScalarFile.read_bounds/2` only decodes the 5°×5° chunk files that overlap the requested bounds, so a state-sized viewport at z=6-7 reads 1-4 chunks instead of a full ~10 MB blob.
- `ScalarFile.read_point/3` reads exactly one chunk (the one containing the snapped cell), so point-detail clicks on a forecast hour no longer cost a whole-file decode.
- The raw `ProfilesFile` is still consulted only when no scalar file exists for `valid_time`, which is the cold-start case.

What's still open:

- During the cold-derive fallback, `ProfilesFile.read/1` still decodes the full file once before the background materializer writes the scalar artifact. The fix is to have the propagation pipeline produce scalar files inline so the fallback path is never taken in production.

### 3. Forecast hours are intentionally not cached, but the real fix is changing what gets read — RESOLVED (in steady state)

Severity: high (was).

What shipped:

- Forecast-hour scrubs no longer require the full ProfilesFile decode + per-cell derivation when a scalar file exists for that hour. `weather_grid_at/2` reads the scalar chunks for the visible viewport directly from NFS.
- Memory budget is unchanged — scalars live on disk, not in ETS, so there's no resident-memory cost per cached forecast hour.

What's still open:

- Same as #1 / #2: the very first scrub to a never-materialized hour pays the cold-derive cost once before the background task fills in the scalar file.

### 4. `GridCache.fetch_bounds/2` still scans the whole cached grid on every pan — RESOLVED

Severity: medium (was).

What shipped:

- `GridCache` now stores each `valid_time` as `%{{lat_band, lon_band} => %{{lat, lon} => row}}` using the same 5°×5° chunk layout as `ScalarFile` ([`grid_cache.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather/grid_cache.ex:30)).
- `fetch_bounds/2` only walks the chunks that intersect the requested viewport, so a state-sized viewport reads ~2-4 chunks instead of filtering ~92k cells.
- `fetch_point/3` reads exactly one chunk.
- All public APIs (`fetch/1`, `fetch_bounds/2`, `fetch_point/3`, `put/2`, `broadcast_put/2`) preserve their pre-rewrite contracts.

### 5. The frontend overlay renderer is doing too much work on the main thread — RESOLVED

Severity: high (was) — the heavy client-side path is gone.

Original symptom:

- `update_weather` rebuilt the lookup map and recreated the overlay every push.
- `GridLayer.createTile` looped over every tile cell on the browser main thread.

What shipped:

- The browser uses a server-rendered `L.tileLayer` pointing at `/weather/tiles/:z/:x/:y` ([`weather_map_hook.ts`](/Users/graham/dev/ntms/microwaveprop/assets/js/weather_map_hook.ts:633)).
- Layer/time changes call `setUrl` + `redraw` rather than rebuilding any data structure.
- Tiles are SVG `<rect>` elements rendered server-side by [`tile_renderer.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop/weather/tile_renderer.ex:148), so per-tile compositing is the browser's tile cache, not Elixir.

What's still open under this finding:

- The server-side tile renderer derives rows from raw `ProfilesFile` data on every tile request — see finding #1 (still the dominant remaining cost).

### 6. The websocket payload is larger than it needs to be — RESOLVED

Severity: medium (was) — the bulk overlay payload is gone.

Original symptom:

- `update_weather` pushed an array of full row maps to the client on every viewport / time change.

What shipped:

- The LiveView no longer pushes weather rows. The only overlay-related events are `update_weather_overlay` and `update_timeline`, which carry just metadata (selected ISO time, list of valid times) ([`weather_map_live.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop_web/live/weather_map_live.ex:106), [`weather_map_live.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop_web/live/weather_map_live.ex:144)).
- Per-cell data is fetched from the tile endpoint by viewport, and point clicks remain a separate small `point_detail` event.

### 6. The websocket payload is larger than it needs to be

Severity: medium

- `update_weather` pushes an array of full row maps to the client: [`weather_map_live.ex`](/Users/graham/dev/ntms/microwaveprop/lib/microwaveprop_web/live/weather_map_live.ex:301).
- Each row contains every weather layer field, even though only one layer is rendered at a time.

Impact:

- Extra server serialization time.
- Extra websocket transfer time.
- Extra browser parsing time before render.

Recommendation:

- Split payloads by use:
  - for map paint, send only `lat`, `lon`, and the selected layer value
  - for click detail, fetch/send the expanded record separately
- If layer switching must be instant without another round trip, use a compact binary or columnar payload for the visible rows instead of verbose JSON objects.
Expected gain:

- Lower end-to-end latency and less client parsing work.

### 7. Point detail has a surprisingly expensive cold path — RESOLVED (in steady state)

Severity: medium (was).

What shipped:

- `weather_point_detail/3` now consults `ScalarFile.read_point/3` between the `GridCache` ETS hit and the legacy `ProfilesFile.read_point/3` path. The scalar lookup decodes a single 5°×5° chunk file, never the full grid.
- For any `valid_time` with a materialized scalar file, a point click is one chunk read + a small list-find — comparable to the in-memory ETS lookup in cost.

What's still open:

- Same as #1: until the propagation pipeline writes scalar files inline, the very first click on a never-materialized forecast hour still falls back to `ProfilesFile.read_point/3`. Mitigated because the bounds-read on the same scrub already kicked off a background materialization.

### 8. Timeline and control rendering — RESOLVED

Severity: low (was).

What shipped:

- The legend `L.Control` is mounted once at `mounted()` and only its inner content (`legendContent.innerHTML`) is patched on layer changes ([`weather_map_hook.ts`](/Users/graham/dev/ntms/microwaveprop/assets/js/weather_map_hook.ts:585)).
- `renderTimeline` keys off the rendered button set (`timelineRenderedKey`). Selection-only updates take an `applyTimelineSelection` patch path that restyles existing buttons in place rather than rebuilding `innerHTML` and re-binding click listeners. A full rebuild only fires when the underlying `timelineData` changes (new forecast hour landed).

## Prioritized Plan

### Phase 1: biggest wins with the least architecture churn — DONE

1. Persist pre-derived weather scalars per `valid_time` instead of deriving them on every `/weather` read. ✅ `Microwaveprop.Weather.ScalarFile`.
2. Persist those scalar artifacts on NFS in a viewport-friendly layout. ✅ 5°×5° chunk files under `<base_dir>/weather_scalars/<iso>/`.
3. Change point-detail reads so one click does not require decoding the whole grid file. ✅ `ScalarFile.read_point/3` reads a single chunk.

Remaining gap: the propagation pipeline doesn't yet write scalar files inline; we lazily materialize on first read with a per-`valid_time` lock. That removes server-side lag for every reader after the first.

### Phase 2: fix the browser-side jank

1. Replace the current `GridLayer.createTile` nested-loop renderer with either:
   - pre-rendered raster tiles, or
   - a single canvas/WebGL overlay backed by numeric arrays.
2. Reduce `update_weather` payloads to the currently selected layer only.

This should remove most pan/zoom/layer-switch lag.

Status:

- Partially complete.
- Done:
  - the old client-side `GridLayer.createTile` path is gone
  - the overlay now loads as viewport-scoped tiles
  - the large `update_weather` row payload is gone for overlay painting
- Still open:
  - tile generation still derives from raw profiles on the server
  - tiles are generated on request rather than read from a weather-specific derived artifact

### Phase 3: improve cache shape for latest-hour panning — DONE

1. Replace full-map `Enum.filter/2` in `GridCache.fetch_bounds/2` with chunked spatial storage. ✅ shipped — `GridCache` now mirrors `ScalarFile`'s 5°×5° chunk layout in ETS, so latest-hour pans walk only the chunks that intersect the viewport.
2. Persist chunks on NFS and read only the visible chunk set. ✅ done in Phase 1.
3. Pre-generate the latest analysis hour and adjacent forecast hours as part of the pipeline so the app serves ready-made artifacts. ✅ shipped — `NotifyListener.handle_propagation_ready/1` fires `Weather.materialize_scalar_file/1` in a detached Task immediately on `NOTIFY propagation_ready`. Moving the derivation into Rust itself remains future work; see Still Pending.

## Suggested Target Architecture

### Write-time

During the propagation/HRRR pipeline, emit three artifacts per `valid_time`:

1. Raw full profile store
   - Used for advanced diagnostics and any future science work.
2. Weather scalar grid store
   - One compact record per grid point containing only final fields used by `/weather`.
   - Persist on NFS.
3. Point-detail index/store
   - Fast random access for one clicked cell.
   - Persist on NFS.
4. Optional raster tile/chunk store
   - Pre-rendered layer assets for the browser.
   - Persist on NFS.

### Read-time

- `/weather` map paint reads scalar grid chunks or raster tiles only.
- NFS is the durable source of truth.
- Point detail reads the indexed point store, not the full grid blob.
- Client receives only what it needs for the active layer or an already-rasterized tile.

## Concrete Implementation Suggestions

### Storage format

- Keep raw `ProfilesFile` as-is for safety during migration.
- Add `WeatherScalarFile` with a compact binary format.
- Store grid points in row-major or chunk-major order with integer indices instead of float map keys.
- Put derived scalar files, point indexes, and optional raster tiles on NFS.

### Server API changes

- Add something like `Weather.scalar_grid_at(valid_time, bounds, layer_id)`.
- Add `Weather.point_detail_at(valid_time, lat, lon)` backed by a point index.
- Keep the current APIs as compatibility wrappers during cutover.
- Prefer chunk-oriented helpers over raw viewport scans, for example:
  - `Weather.visible_chunks(bounds)`
  - `Weather.chunk_rows_at(valid_time, layer_id, chunk_ids)`

### Frontend changes

- Replace `gridLookup: Map<string, WeatherPoint>` with indexed numeric buffers.
- Avoid rebuilding the overlay object unless the viewport or layer renderer truly changes.
- Prefer one draw pass per visible grid rather than per-tile nested string lookups.
- If raster tiles are adopted, the frontend becomes much simpler:
  - Leaflet swaps tile URLs by `valid_time` + `layer_id`
  - the websocket coordinates only state and point-detail interactions

## More Optimal Storage / Retrieval Options

These are the realistic options, from least to most optimized for `/weather`.

### Option A: keep the current raw profile blob model

Description:

- Store one full raw profile file per `valid_time`.
- Derive rows on reads.

Pros:

- Minimal write-path change.

Cons:

- Worst read performance.
- Wrong shape for viewport reads and point-detail reads.
- Repeats expensive atmospheric derivation work.

Verdict:

- Not recommended for `/weather`.

### Option B: one derived scalar file per `valid_time`

Description:

- At write time, derive the final weather fields once.
- Store one compact scalar file per hour on NFS.

Pros:

- Much better than current design.
- Simplest migration path.

Cons:

- Still requires full-file decode unless you add an index.

Verdict:

- Good intermediate step.

### Option C: chunked derived scalar files per `valid_time`

Description:

- Store weather rows grouped into spatial chunks on NFS.
- Read only visible chunks.

Pros:

- Best fit for viewport-driven map access.
- Good memory behavior.
- Works well for both JSON and raster generation paths.

Cons:

- Slightly more write-path complexity.
- Requires chunk manifest management.

Verdict:

- Best overall backend format for this app.

### Option D: pre-rendered raster tiles per `valid_time` and layer

Description:

- Generate tile images or larger raster chunks for each weather layer.
- Browser reads tiles directly; websocket traffic drops sharply.

Pros:

- Best browser responsiveness.
- Simplest client rendering path.
- Excellent for many concurrent viewers.

Cons:

- More storage.
- More write-time fanout across layers.
- Point detail still needs a scalar/point store beside the tiles.

Verdict:

- Best frontend performance, especially if `/weather` is primarily a visual browsing tool.

### Recommended Combination

If the goal is the best practical design without adding more cache infrastructure:

1. Store chunked derived scalar files on NFS.
2. Store a point-detail index/store on NFS.
3. Optionally add pre-rendered raster tiles on NFS for the most-used layers once the scalar chunk path is stable.

That gives you:

- efficient write-once derivation
- fast viewport reads
- fast point clicks
- predictable memory use
- a clean path to even faster raster rendering later

## Best-Fit Scaled-Down Solution

If I were implementing this here, I would choose:

1. Raw profiles remain in `ProfilesFile` for science/debug paths.
2. A new NFS-backed chunked `WeatherScalarFile` becomes the primary `/weather` read path.
3. A new NFS-backed point index/store becomes the primary point-detail read path.
4. Later, if `/weather` still needs more smoothness, layer rasters/tiles get generated from the scalar chunks and served directly.

That is the most balanced scaled-down design here. It is materially faster than the current shape, avoids wasting CPU on repeated derivation, and does not depend on large memory-resident caches.

## Bottom Line

All listed phases are shipped:

- `/weather` reads from pre-derived scalar chunks on NFS instead of re-deriving from raw HRRR data on every request.
- Viewport reads only touch the chunks that overlap the current bounds — for both NFS-backed `ScalarFile` reads and the in-memory `GridCache`.
- Point clicks read a single chunk instead of the whole grid file.
- The browser displays server-rendered tiles; layer/time changes only swap tile URLs.
- New forecast hours land with the scalar artifact already on disk: the Rust pipeline writes scalar chunks inline alongside the raw profile, and the Elixir-side `NotifyListener.handle_propagation_ready/1` keeps an idempotent fallback materialization for hours that pre-date the Rust deploy.
- Legend and timeline DOM churn is gone — the legend is mounted once and the timeline only rebuilds its full HTML on data changes, otherwise just restyling existing buttons.

Both languages produce and consume identical bytes on disk: gzipped MessagePack chunks under `<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.mp.gz`. There is no Elixir-only or Rust-only branch in the format.
