MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer, AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime). Net effect: 30 GRIB2 decodes/hour of dead work on hot pods. Removed: - MrmsFetchWorker, MrmsClient, MrmsCache and their tests - cron entries in config.exs, dev.exs, runtime.exs - mrms_req_options stub in test.exs - MrmsCache from supervision tree in application.ex - stale MRMS references in PropagationGridWorker docstring and test Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port) follow-on work.
607 lines
24 KiB
Markdown
607 lines
24 KiB
Markdown
# Rust Migration Phase 3 — f00 Port + MRMS Deletion + HRRR Per-QSO Port
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Remove every remaining GRIB2 decode from BEAM so hot pods can shrink to 1 Gi, and delete one dead pipeline (MRMS) along the way.
|
||
|
||
**Architecture:** Three independent work streams against the same shared `prop-grid-rs` binary. Stream A ports f00's native-duct + NEXRAD + commercial-link merge from Elixir into Rust via a new `grid_tasks.kind` column so the same work queue handles both forecast hours and analysis-hour enrichment. Stream B deletes the MRMS worker/cache/client entirely (the only consumer `AsosAdjustmentWorker` is already disabled). Stream C ports the per-QSO `HrrrFetchWorker` into a new Rust binary sharing the same crate, taking the `:hrrr` queue off BEAM.
|
||
|
||
**Tech Stack:** Rust (tokio, sqlx, reqwest, wgrib2 subprocess), Elixir (Oban, Postgrex.Notifications), Postgres (grid_tasks work queue + LISTEN/NOTIFY), Kubernetes/flux.
|
||
|
||
---
|
||
|
||
## Work Streams Overview
|
||
|
||
| Stream | Priority | What | Payoff |
|
||
|---|---|---|---|
|
||
| **A: Port f00** | 1 | Move f00 chain step (native duct + NEXRAD + commercial + ProfilesFile write) from Elixir to Rust. | Finishes hourly-chain memory story. Hot pod can drop from 2Gi/replica × N replicas to 1Gi. |
|
||
| **B: Delete MRMS** | 2 | Remove MrmsFetchWorker, MrmsClient, MrmsCache, and the `*/2 * * * *` cron. | Removes 30 GRIB2 decodes/hour of dead work. Zero risk — AsosAdjustmentWorker (only consumer) is already disabled. |
|
||
| **C: Port HrrrFetchWorker** | 3 | Move per-QSO HRRR profile fetch to Rust. New `hrrr_point_tasks` queue. Elixir enqueues, Rust fulfills. | Takes the `:hrrr` queue off BEAM. Removes per-QSO refc-binary churn and the explicit `:erlang.garbage_collect` that's papering over it. |
|
||
|
||
Each stream is independently mergeable and reversible. Recommended order: **B → A → C**. B first because it's a pure delete (simplest; shrinks the attack surface before A touches more code). A second because it's the real memory win. C last because it's optional polish.
|
||
|
||
---
|
||
|
||
## Stream B: Delete MRMS (start here — warm-up)
|
||
|
||
**Context:** `MrmsFetchWorker` runs every 2 minutes on the `:propagation` queue. It decodes an MRMS PrecipRate GRIB2 and caches the regridded result in `MrmsCache`. The only consumer that reads that cache, `AsosAdjustmentWorker`, is disabled in all three configs (see `config/runtime.exs:212-216`). Net effect: 30 GRIB2 decodes/hour of dead work on the hot pods.
|
||
|
||
### Task B1: Confirm no live consumers
|
||
|
||
**Files to check:**
|
||
- Search for `MrmsCache` and `Mrms\.` across `lib/` and `test/` — confirm only the worker/client/cache/tests reference it.
|
||
- Search for `AsosAdjustmentWorker` — confirm the only references are disabled cron entries + the worker file itself.
|
||
|
||
**Step 1: Run the searches**
|
||
|
||
```bash
|
||
grep -rn "MrmsCache\|Mrms\." lib/ config/ | grep -v "#"
|
||
grep -rn "AsosAdjustmentWorker" config/ lib/
|
||
```
|
||
|
||
**Expected output:** References only in `workers/mrms_fetch_worker.ex`, `weather/mrms_client.ex`, `weather/mrms_cache.ex`, `application.ex` (supervisor), and commented-out cron lines. No LiveView, context, or worker depends on it.
|
||
|
||
**If you find a live consumer, STOP** — add that consumer to the removal list before proceeding.
|
||
|
||
### Task B2: Remove MRMS from supervisor + cron
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/application.ex` (drop `MrmsCache` from the supervision tree)
|
||
- Modify: `config/runtime.exs:206-211` (drop the `MrmsFetchWorker` cron line and its comment block)
|
||
- Modify: `config/config.exs:101` (same removal in dev-carrying config)
|
||
- Modify: `config/dev.exs:100` (same)
|
||
|
||
**Step 1: Write the failing test**
|
||
|
||
Add to `test/microwaveprop/application_test.exs` (create if missing):
|
||
|
||
```elixir
|
||
defmodule Microwaveprop.ApplicationTest do
|
||
use ExUnit.Case, async: true
|
||
|
||
test "MrmsCache is not in the supervision tree" do
|
||
children = Microwaveprop.Application.children_for_env(:test)
|
||
refute Enum.any?(children, &match?({Microwaveprop.Weather.MrmsCache, _}, &1))
|
||
refute Enum.any?(children, fn
|
||
Microwaveprop.Weather.MrmsCache -> true
|
||
_ -> false
|
||
end)
|
||
end
|
||
end
|
||
```
|
||
|
||
If `children_for_env/1` doesn't exist, either extract it from `start/2` or inline the list in the test.
|
||
|
||
**Step 2: Run test to verify it fails**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/application_test.exs
|
||
```
|
||
|
||
Expected: FAIL — MrmsCache still in supervision tree.
|
||
|
||
**Step 3: Remove MrmsCache from supervision tree**
|
||
|
||
Edit `lib/microwaveprop/application.ex`, delete the line that adds `MrmsCache`.
|
||
|
||
**Step 4: Run test to verify it passes**
|
||
|
||
```bash
|
||
mix test test/microwaveprop/application_test.exs
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Remove cron entries**
|
||
|
||
Delete the `MrmsFetchWorker` entry + its leading comment block from all three config files.
|
||
|
||
**Step 6: Commit**
|
||
|
||
```bash
|
||
git add lib/microwaveprop/application.ex config/ test/microwaveprop/application_test.exs
|
||
git commit -m "chore(mrms): remove from supervision tree and cron (consumer disabled)"
|
||
```
|
||
|
||
### Task B3: Delete MRMS source files and tests
|
||
|
||
**Files:**
|
||
- Delete: `lib/microwaveprop/workers/mrms_fetch_worker.ex`
|
||
- Delete: `lib/microwaveprop/weather/mrms_client.ex`
|
||
- Delete: `lib/microwaveprop/weather/mrms_cache.ex`
|
||
- Delete: `test/microwaveprop/workers/mrms_fetch_worker_test.exs`
|
||
- Delete: `test/microwaveprop/weather/mrms_cache_test.exs`
|
||
- Modify: `config/test.exs:66` (drop `mrms_req_options`)
|
||
|
||
**Step 1: Delete files**
|
||
|
||
```bash
|
||
rm lib/microwaveprop/workers/mrms_fetch_worker.ex \
|
||
lib/microwaveprop/weather/mrms_client.ex \
|
||
lib/microwaveprop/weather/mrms_cache.ex \
|
||
test/microwaveprop/workers/mrms_fetch_worker_test.exs \
|
||
test/microwaveprop/weather/mrms_cache_test.exs
|
||
```
|
||
|
||
**Step 2: Drop the test config line**
|
||
|
||
Edit `config/test.exs:66`, remove the `mrms_req_options` line.
|
||
|
||
**Step 3: Run full test suite**
|
||
|
||
```bash
|
||
mix precommit
|
||
```
|
||
|
||
Expected: clean compile (warnings-as-errors), all tests pass. If `mix xref` surfaces a caller, fix it.
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add -A
|
||
git commit -m "chore(mrms): delete worker, client, cache, and tests"
|
||
```
|
||
|
||
### Task B4: Drop MRMS k8s resources (if any)
|
||
|
||
**Files to inspect:**
|
||
- `k8s/kustomization.yaml`, any `mrms_*` resource
|
||
|
||
**Step 1: Check**
|
||
|
||
```bash
|
||
grep -r "mrms\|Mrms" k8s/
|
||
```
|
||
|
||
**Step 2: If found, remove. Otherwise skip.**
|
||
|
||
**Step 3: Commit if changed.**
|
||
|
||
---
|
||
|
||
## Stream A: Port f00 to Rust
|
||
|
||
**Context:** Today, Elixir's `PropagationGridWorker.perform/1` handles f00 only — the analysis-hour step that does three things Rust doesn't: native-level HRRR duct merge (the ~530 MB native GRIB2), NEXRAD composite reflectivity overlay, and commercial-link degradation lookup. It also writes the `ProfilesFile` that `/weather` reads. Forecast hours f01–f18 already run in Rust via the `grid_tasks` queue.
|
||
|
||
**Strategy:** Add a `kind` column to `grid_tasks` so the same queue serves both forecast hours (`kind='forecast'`) and analysis-hour enrichment (`kind='analysis'`). Rust's worker dispatches on `kind`. Elixir's seeder switches from `Oban.insert_all` for f00 to seeding a `kind='analysis'` row alongside the 18 `kind='forecast'` rows. When the analysis task completes, Rust writes the same `/data/scores/<band>/<iso>.ntms` + `/data/profiles/<iso>.ntms` files and NOTIFYs `propagation_ready` with the existing payload.
|
||
|
||
### Task A1: Add `kind` column to grid_tasks
|
||
|
||
**Files:**
|
||
- Create: `priv/repo/migrations/YYYYMMDDHHMMSS_add_kind_to_grid_tasks.exs`
|
||
|
||
**Step 1: Generate migration**
|
||
|
||
```bash
|
||
mix ecto.gen.migration add_kind_to_grid_tasks
|
||
```
|
||
|
||
**Step 2: Write the migration**
|
||
|
||
```elixir
|
||
defmodule Microwaveprop.Repo.Migrations.AddKindToGridTasks do
|
||
use Ecto.Migration
|
||
|
||
def change do
|
||
alter table(:grid_tasks) do
|
||
add :kind, :string, null: false, default: "forecast"
|
||
end
|
||
|
||
create index(:grid_tasks, [:status, :kind])
|
||
|
||
# Replace the old unique index with one that includes kind so an
|
||
# analysis task and a forecast task for the same (run_time, f00)
|
||
# can coexist without conflict.
|
||
drop_if_exists unique_index(:grid_tasks, [:run_time, :forecast_hour])
|
||
create unique_index(:grid_tasks, [:run_time, :forecast_hour, :kind])
|
||
end
|
||
end
|
||
```
|
||
|
||
**Step 3: Run migration**
|
||
|
||
```bash
|
||
mix ecto.migrate
|
||
```
|
||
|
||
**Step 4: Commit**
|
||
|
||
```bash
|
||
git add priv/repo/migrations/
|
||
git commit -m "feat(grid_tasks): add kind column to separate forecast vs analysis work"
|
||
```
|
||
|
||
### Task A2: Rust reads `kind` and routes
|
||
|
||
**Files:**
|
||
- Modify: `rust/prop_grid_rs/src/db.rs` — add `kind: String` to the task struct, select it in `claim_next`
|
||
- Modify: `rust/prop_grid_rs/src/bin/worker.rs` — match on `kind` and dispatch to forecast vs. analysis pipeline
|
||
- Test: `rust/prop_grid_rs/src/db.rs` test for `claim_next` returning the `kind`
|
||
|
||
**Step 1: Write failing test**
|
||
|
||
In `rust/prop_grid_rs/src/db.rs`:
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn claim_next_returns_kind() {
|
||
// seed one task with kind='analysis', claim it, assert kind round-trips
|
||
// ...
|
||
}
|
||
```
|
||
|
||
**Step 2: Run test**
|
||
|
||
```bash
|
||
cd rust/prop_grid_rs && cargo test claim_next_returns_kind
|
||
```
|
||
|
||
Expected: FAIL (field doesn't exist).
|
||
|
||
**Step 3: Add `kind` field to the claim struct + SELECT list**
|
||
|
||
Minimal change to `claim_next` to read and return `kind`. Default to `"forecast"` if unmapped so the compile stays green during rollout.
|
||
|
||
**Step 4: Run test**
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add rust/prop_grid_rs/src/db.rs
|
||
git commit -m "feat(prop-grid-rs): surface grid_tasks.kind in claim_next"
|
||
```
|
||
|
||
### Task A3: Port `HrrrNativeClient.fetch_native_duct_grid` to Rust
|
||
|
||
This is the biggest subtask — the 530 MB GRIB2 native-level pull + duct-metric computation.
|
||
|
||
**Files:**
|
||
- Create: `rust/prop_grid_rs/src/native_duct.rs`
|
||
- Test: `rust/prop_grid_rs/src/native_duct.rs` (inline `#[cfg(test)] mod tests`)
|
||
|
||
**Step 1: Write golden-fixture test**
|
||
|
||
Follow the same pattern Phase 1 used for surface/pressure: a Mix task dumps the Elixir-computed duct grid for a known `(date, hour)` to a bincode file in `rust/prop_grid_rs/tests/golden/`. Rust loads it and asserts exact match on every `{lat, lon}` key and per-field values.
|
||
|
||
Create `lib/mix/tasks/rust.golden_duct.ex`:
|
||
|
||
```elixir
|
||
defmodule Mix.Tasks.Rust.GoldenDuct do
|
||
use Mix.Task
|
||
|
||
@shortdoc "Dump Elixir-computed native duct grid for Rust comparison"
|
||
|
||
def run([date_str, hour_str]) do
|
||
Mix.Task.run("app.start")
|
||
date = Date.from_iso8601!(date_str)
|
||
hour = String.to_integer(hour_str)
|
||
grid_spec = Microwaveprop.Propagation.Grid.wgrib2_grid_spec()
|
||
|
||
{:ok, duct_grid} =
|
||
Microwaveprop.Weather.HrrrNativeClient.fetch_native_duct_grid(date, hour, grid_spec, 0)
|
||
|
||
path = "rust/prop_grid_rs/tests/golden/duct_#{date_str}_t#{hour_str}z.bincode"
|
||
File.mkdir_p!(Path.dirname(path))
|
||
File.write!(path, :erlang.term_to_binary(duct_grid))
|
||
IO.puts("Wrote #{map_size(duct_grid)} cells to #{path}")
|
||
end
|
||
end
|
||
```
|
||
|
||
**Step 2: Run the golden dump against a recent hour**
|
||
|
||
```bash
|
||
mix rust.golden_duct 2026-04-19 14
|
||
```
|
||
|
||
Expected: writes a bincode file with ~92k cells.
|
||
|
||
**Step 3: Write the Rust failing test**
|
||
|
||
In `rust/prop_grid_rs/src/native_duct.rs`:
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn matches_elixir_golden() {
|
||
let golden = load_golden("tests/golden/duct_2026-04-19_t14z.bincode");
|
||
let rust = fetch_native_duct_grid(Date::from_ymd(2026, 4, 19), 14, grid_spec(), 0).await.unwrap();
|
||
|
||
assert_eq!(rust.len(), golden.len());
|
||
for (pt, expected) in &golden {
|
||
let got = rust.get(pt).expect(&format!("missing {:?}", pt));
|
||
assert!((got.native_min_gradient - expected.native_min_gradient).abs() < 1e-3);
|
||
assert_eq!(got.duct_count, expected.duct_count);
|
||
// ... other fields
|
||
}
|
||
}
|
||
```
|
||
|
||
**Step 4: Run test**
|
||
|
||
```bash
|
||
cargo test matches_elixir_golden
|
||
```
|
||
|
||
Expected: FAIL — `fetch_native_duct_grid` doesn't exist.
|
||
|
||
**Step 5: Implement the Rust port**
|
||
|
||
Port these pieces in order, one commit each:
|
||
|
||
1. **Byte-range fetch for duct variables.** Port `duct_byte_ranges/1` (selects TMP/SPFH/HGT/PRES on all 50 hybrid levels from the idx). Reuse the existing `fetcher::download_ranges_to_file` that Phase 1 built.
|
||
2. **wgrib2 subprocess for `extract_grid_from_file_mapped`.** Port the `-lola` + match-pattern invocation. Parse the flat output into a `HashMap<(lat, lon), HashMap<String, f32>>`. Stream cell-by-cell rather than buffering all values — matches Elixir's "peak memory ~86 MB" design.
|
||
3. **`compute_duct_metrics` per cell.** Port `build_native_profile/1`, `min_m_gradient/1`, and `Duct.analyze/1`. `Duct.analyze/1` is in `lib/microwaveprop/propagation/duct.ex` — that's its own port.
|
||
4. **Wire into `native_duct.rs::fetch_native_duct_grid`** as the public entry point.
|
||
|
||
**Step 6: Run test to verify it passes**
|
||
|
||
```bash
|
||
cargo test matches_elixir_golden
|
||
```
|
||
|
||
Expected: PASS with max-per-cell delta under 1e-3.
|
||
|
||
**Step 7: Commit**
|
||
|
||
```bash
|
||
git add rust/prop_grid_rs/src/native_duct.rs rust/prop_grid_rs/tests/golden/ lib/mix/tasks/rust.golden_duct.ex
|
||
git commit -m "feat(prop-grid-rs): port HrrrNativeClient.fetch_native_duct_grid"
|
||
```
|
||
|
||
### Task A4: Port NEXRAD composite reflectivity fetch
|
||
|
||
**Context:** `NexradClient.fetch_frame/2` (in `lib/microwaveprop/weather/nexrad_client.ex`) pulls the current MRMS composite reflectivity mosaic and returns `[%{lat, lon, max_reflectivity_dbz}]`. The merge into `grid_data` is line-level (`apply_nexrad_observations`). It's another GRIB2 decode but much smaller than the native-level file.
|
||
|
||
**Files:**
|
||
- Create: `rust/prop_grid_rs/src/nexrad.rs`
|
||
- Test: inline fixture test
|
||
|
||
**Step 1: Golden fixture**
|
||
|
||
Same pattern: `mix rust.golden_nexrad 2026-04-19T14:00:00Z` dumps the Elixir observations list as bincode.
|
||
|
||
**Step 2: Write failing test**
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn matches_elixir_golden() { /* ... */ }
|
||
```
|
||
|
||
**Step 3: Implement `fetch_frame`** by porting the HTTP GET + GRIB2 decode from `NexradClient`.
|
||
|
||
**Step 4: Run test.** Expected PASS.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
### Task A5: Port commercial-link degradation lookup
|
||
|
||
**Context:** `Commercial.build_link_lookup/1` (in `lib/microwaveprop/commercial.ex`) returns a `[{{lat, lon}, degradation}]` list from Postgres. `link_degradation_from_lookup/2` is a pure haversine match. Only 7 links cluster around DFW so most cells return nil.
|
||
|
||
**Files:**
|
||
- Create: `rust/prop_grid_rs/src/commercial.rs`
|
||
- Test: inline SQL + haversine round-trip
|
||
|
||
**Step 1: Golden fixture (simpler — just dump the 7 link rows)**
|
||
|
||
**Step 2: Failing test**
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn haversine_matches_elixir() { /* 7 links × 10 sample points */ }
|
||
```
|
||
|
||
**Step 3: Port `build_link_lookup` (one SELECT) + haversine matcher.**
|
||
|
||
**Step 4: Commit.**
|
||
|
||
### Task A6: Port ProfilesFile write
|
||
|
||
**Context:** `Microwaveprop.Propagation.ProfilesFile.write!/2` serializes the fully-enriched `grid_data` to `/data/profiles/<iso>.ntms` using `:erlang.term_to_binary/1`. This is the tricky one — the file format is Erlang ETF.
|
||
|
||
**Decision point:** Two options.
|
||
|
||
1. **Port the reader to Rust + ETF encoder.** The `eetf` crate exists but is not heavily maintained. Risk: ETF subtleties (ref counting, atom tables) could produce bit-different output that breaks Elixir's reader.
|
||
2. **Change the format.** Replace ETF with a simple framed bincode/cbor/msgpack file. Rust writes, Elixir reads via a new decoder. Clean break.
|
||
|
||
**Recommendation: Option 2.** The profiles file is read from exactly one place (`/weather` and point-detail LiveView in `lib/microwaveprop/weather.ex`). Porting the reader to a new format is ~50 lines. Fighting with eetf is a tarpit.
|
||
|
||
**Files:**
|
||
- Create: `rust/prop_grid_rs/src/profiles_file.rs` (writes new format)
|
||
- Modify: `lib/microwaveprop/propagation/profiles_file.ex` (reads old format; add reader for new format; write-path deleted once Rust owns f00)
|
||
- Modify: `lib/microwaveprop/weather.ex` (call sites that read profiles)
|
||
|
||
**Step 1: Design the format.** Header: magic `PROF\x00`, version u8, record count u32, `{lat: f32, lon: f32, field_count: u8, [{key: varint-len-prefixed string, value: tagged-union}]...}`. Or — simpler — MessagePack. `rmp-serde` in Rust, `Msgpax` in Elixir. Both are well-maintained.
|
||
|
||
**Step 2: Write Elixir reader for MessagePack variant.** Keep the old ETF reader around under a `read_etf!/1` alias for backward compatibility during rollout.
|
||
|
||
**Step 3: Failing test in Rust.** Round-trip: write a sample `grid_data`, read it back, assert equal.
|
||
|
||
**Step 4: Implement writer.**
|
||
|
||
**Step 5: Dual-read test in Elixir.** Old ETF files still read; new MessagePack files read identically.
|
||
|
||
**Step 6: Commit.**
|
||
|
||
### Task A7: Wire analysis pipeline in Rust worker
|
||
|
||
**Files:**
|
||
- Modify: `rust/prop_grid_rs/src/pipeline.rs` — add `run_analysis_task` alongside the existing forecast pipeline
|
||
- Modify: `rust/prop_grid_rs/src/bin/worker.rs` — dispatch on `task.kind`
|
||
|
||
**Step 1: Write failing integration test**
|
||
|
||
In `rust/prop_grid_rs/tests/analysis_pipeline_test.rs`:
|
||
|
||
```rust
|
||
#[tokio::test]
|
||
async fn analysis_task_writes_scores_and_profiles() {
|
||
// seed one kind='analysis' task, run pipeline, assert
|
||
// /data/scores/10000/<iso>.ntms exists
|
||
// /data/profiles/<iso>.ntms exists
|
||
// NOTIFY propagation_ready fired
|
||
}
|
||
```
|
||
|
||
**Step 2: Run test.** Expected FAIL.
|
||
|
||
**Step 3: Implement `run_analysis_task`.** Same shape as forecast pipeline but:
|
||
- Calls `native_duct::fetch_native_duct_grid` after the regular surface+pressure merge.
|
||
- Calls `nexrad::fetch_frame` and merges.
|
||
- Calls `commercial::build_link_lookup` and merges.
|
||
- Writes `ProfilesFile` to `/data/profiles/<iso>.ntms`.
|
||
- Scores all bands as before.
|
||
|
||
**Step 4: Run test.** Expected PASS.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
### Task A8: Flip Elixir seeder to enqueue analysis as grid_task
|
||
|
||
**Files:**
|
||
- Modify: `lib/microwaveprop/workers/propagation_grid_worker.ex:80-99` (seed_chain)
|
||
- Modify: `lib/microwaveprop/propagation/grid_task_enqueuer.ex` (add `seed_with_analysis/1`)
|
||
|
||
**Step 1: Write failing test**
|
||
|
||
In `test/microwaveprop/workers/propagation_grid_worker_test.exs`, update the seed test to assert:
|
||
- **Zero** Oban jobs enqueued (f00 is no longer an Oban job).
|
||
- **19** grid_tasks rows: 1 with `kind='analysis' AND forecast_hour=0`, 18 with `kind='forecast' AND forecast_hour IN (1..18)`.
|
||
|
||
**Step 2: Run test.** Expected FAIL.
|
||
|
||
**Step 3: Modify `seed_chain` to call `GridTaskEnqueuer.seed_with_analysis(run_time)` and return `:ok`.** Delete the `Oban.insert_all` line. Delete `run_chain_step`, `process_forecast_hour`, and all the merge helpers from `propagation_grid_worker.ex`. The module becomes a thin cron handler.
|
||
|
||
**Step 4: Run test.** Expected PASS.
|
||
|
||
**Step 5: Delete now-unused Elixir modules** after `mix xref callers` confirms no callers remain:
|
||
- `lib/microwaveprop/weather/hrrr_native_client.ex`
|
||
- `lib/microwaveprop/weather/nexrad_client.ex` (assuming no other caller)
|
||
- The `merge_native_duct_data`, `merge_nexrad_data`, `merge_commercial_link_data`, `compute_scores_*` helpers
|
||
|
||
**Step 6: Commit.**
|
||
|
||
### Task A9: Shrink hot pod memory limit
|
||
|
||
**Files:**
|
||
- Modify: `k8s/deployment.yaml`
|
||
|
||
**Step 1: Wait for 24h of clean post-cutover runs.**
|
||
|
||
**Step 2: Reduce memory limit from 2Gi to 1Gi, requests from 512Mi to 256Mi.**
|
||
|
||
**Step 3: Watch `kubectl -n prop top pod` for OOM signals over 24h.** If any pod OOMKills, revert immediately and investigate before retrying.
|
||
|
||
**Step 4: Commit.**
|
||
|
||
---
|
||
|
||
## Stream C: Port HrrrFetchWorker (optional polish)
|
||
|
||
**Context:** Each QSO submission enqueues an `HrrrFetchWorker` job that fetches a HRRR profile at the contact location. Per-job payload is small (one point, ~100 kB of GRIB2 bytes) but the job calls `HrrrClient` which on BEAM holds refc binaries that the worker explicitly `:erlang.garbage_collect`s at the end. That GC is a signal the binary pressure is real.
|
||
|
||
**Strategy:** New Rust binary `hrrr-point-worker` shares the existing `prop_grid_rs` crate (adds a second bin). A new `hrrr_point_tasks` table mirrors `grid_tasks`. Elixir's ingestion switches from `HrrrFetchWorker` (Oban) to inserting into `hrrr_point_tasks`.
|
||
|
||
### Task C1: Migration for `hrrr_point_tasks`
|
||
|
||
**Files:**
|
||
- Create: `priv/repo/migrations/YYYYMMDDHHMMSS_create_hrrr_point_tasks.exs`
|
||
|
||
**Columns:** `id uuid PK`, `qso_id uuid`, `valid_time timestamp`, `lat double precision`, `lon double precision`, `status text`, `claimed_at`, `completed_at`, `error text`, `attempt smallint`. Unique on `(qso_id, valid_time)`.
|
||
|
||
**Step 1: Generate + write migration.**
|
||
**Step 2: Run migration.**
|
||
**Step 3: Commit.**
|
||
|
||
### Task C2: Rust `hrrr-point-worker` binary
|
||
|
||
**Files:**
|
||
- Create: `rust/prop_grid_rs/src/bin/hrrr_point_worker.rs`
|
||
- Create: `rust/prop_grid_rs/src/point_fetcher.rs` (per-point HRRR profile extract)
|
||
|
||
**Step 1: Golden fixture for HrrrClient.fetch_profile.**
|
||
**Step 2: Failing test.**
|
||
**Step 3: Implement point fetcher + worker loop.** Reuse the existing `fetcher` module for byte-range downloads; add a new `extract_point_profile` that takes a single `(lat, lon)` instead of a grid spec.
|
||
**Step 4: Commit.**
|
||
|
||
### Task C3: k8s deployment for hrrr-point-worker
|
||
|
||
**Files:**
|
||
- Create: `k8s/deployment-hrrr-point-rs.yaml` (same image, different ENTRYPOINT)
|
||
- Modify: `k8s/kustomization.yaml`
|
||
|
||
**Step 1: Write manifest. 1 replica on talos5 initially, 256 Mi memory.**
|
||
**Step 2: Apply.**
|
||
**Step 3: Commit.**
|
||
|
||
### Task C4: Switch Elixir ingestion to enqueue into `hrrr_point_tasks`
|
||
|
||
**Files:**
|
||
- Modify: wherever `HrrrFetchWorker.new(...)` is called (check `lib/microwaveprop/qsos.ex` or `lib/microwaveprop/radio.ex` via `grep -rn "HrrrFetchWorker.new"`)
|
||
- Delete: `lib/microwaveprop/workers/hrrr_fetch_worker.ex` (after `mix xref callers` is clean)
|
||
- Delete: corresponding test
|
||
|
||
**Step 1: Failing test.** QSO submission inserts a row into `hrrr_point_tasks` and does NOT enqueue an Oban job.
|
||
**Step 2: Implement.**
|
||
**Step 3: Commit.**
|
||
|
||
---
|
||
|
||
## Critical Files Reference
|
||
|
||
**Ported out (deleted at end of Stream A):**
|
||
- `lib/microwaveprop/weather/hrrr_native_client.ex`
|
||
- `lib/microwaveprop/weather/nexrad_client.ex` (confirm no per-QSO callers first)
|
||
- `lib/microwaveprop/weather/mrms_client.ex` (Stream B)
|
||
- `lib/microwaveprop/weather/mrms_cache.ex` (Stream B)
|
||
- `lib/microwaveprop/workers/mrms_fetch_worker.ex` (Stream B)
|
||
- `lib/microwaveprop/workers/hrrr_fetch_worker.ex` (Stream C)
|
||
- Most of `lib/microwaveprop/workers/propagation_grid_worker.ex` — becomes a thin seeder
|
||
|
||
**Kept as-is:**
|
||
- `lib/microwaveprop/commercial.ex` — Elixir still needs this for the SNMP poller and LiveView sensor widget. Rust just adds a parallel reader over the same `commercial_samples` table.
|
||
- `lib/microwaveprop/propagation/duct.ex` — ported to Rust, original stays because other callers (per-contact analysis) still use it.
|
||
|
||
**New Rust modules (Stream A):**
|
||
- `rust/prop_grid_rs/src/native_duct.rs`
|
||
- `rust/prop_grid_rs/src/nexrad.rs`
|
||
- `rust/prop_grid_rs/src/commercial.rs`
|
||
- `rust/prop_grid_rs/src/profiles_file.rs`
|
||
- `rust/prop_grid_rs/src/duct.rs` (ported from `lib/microwaveprop/propagation/duct.ex`)
|
||
|
||
**New Rust module (Stream C):**
|
||
- `rust/prop_grid_rs/src/point_fetcher.rs`
|
||
- `rust/prop_grid_rs/src/bin/hrrr_point_worker.rs`
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
**Stream B:**
|
||
- `grep -r "Mrms" lib/ test/ config/` returns nothing.
|
||
- `mix precommit` green.
|
||
- 24h later: `kubectl -n prop get events` shows no MrmsFetchWorker executions.
|
||
|
||
**Stream A:**
|
||
- `cargo test -p prop_grid_rs` green with golden-duct, golden-nexrad, golden-commercial fixtures.
|
||
- `mix test` green.
|
||
- One full post-cutover hourly cycle shows `kind='analysis'` task `status='done'` in `grid_tasks`, `/data/profiles/<iso>.ntms` written, `/weather` page loads correctly.
|
||
- Hot pod `beam_memory_total` < 700 MiB peak over 24 h (down from ~1 GiB under Phase 2).
|
||
- Hot pod memory limit reduced to 1 Gi with zero OOMKilled events over 7 days.
|
||
|
||
**Stream C:**
|
||
- QSO submission inserts a `hrrr_point_tasks` row within 1 s.
|
||
- Rust `hrrr-point-worker` logs show claim→complete within 30 s of insert.
|
||
- HRRR profile appears on the QSO detail page within 1 min.
|
||
- `:hrrr` Oban queue is empty and can be removed from the Oban config.
|
||
|
||
---
|
||
|
||
## Rollback Plans
|
||
|
||
**Stream B:** `git revert` restores MRMS. Supervision tree re-adds MrmsCache on next deploy. No data loss — MrmsCache was in-memory only.
|
||
|
||
**Stream A:** `git revert` restores the Elixir f00 chain step. The `grid_tasks.kind` column can stay (default `'forecast'` keeps existing Rust behaviour). Bump hot pod memory back to 2Gi if Phase 3 shrink was already deployed.
|
||
|
||
**Stream C:** `git revert` restores `HrrrFetchWorker`. `hrrr_point_tasks` rows become orphan; a one-off Mix task migrates them back to Oban jobs.
|