prop/docs/plans/2026-04-19-rust-migration-phase3.md
Graham McIntire 48708621c5
chore(mrms): delete pipeline — consumer AsosAdjustmentWorker already disabled
MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an
MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer,
AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime).
Net effect: 30 GRIB2 decodes/hour of dead work on hot pods.

Removed:
- MrmsFetchWorker, MrmsClient, MrmsCache and their tests
- cron entries in config.exs, dev.exs, runtime.exs
- mrms_req_options stub in test.exs
- MrmsCache from supervision tree in application.ex
- stale MRMS references in PropagationGridWorker docstring and test

Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks
the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port)
follow-on work.
2026-04-19 17:25:59 -05:00

24 KiB
Raw Permalink Blame History

Rust Migration Phase 3 — f00 Port + MRMS Deletion + HRRR Per-QSO Port

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Remove every remaining GRIB2 decode from BEAM so hot pods can shrink to 1 Gi, and delete one dead pipeline (MRMS) along the way.

Architecture: Three independent work streams against the same shared prop-grid-rs binary. Stream A ports f00's native-duct + NEXRAD + commercial-link merge from Elixir into Rust via a new grid_tasks.kind column so the same work queue handles both forecast hours and analysis-hour enrichment. Stream B deletes the MRMS worker/cache/client entirely (the only consumer AsosAdjustmentWorker is already disabled). Stream C ports the per-QSO HrrrFetchWorker into a new Rust binary sharing the same crate, taking the :hrrr queue off BEAM.

Tech Stack: Rust (tokio, sqlx, reqwest, wgrib2 subprocess), Elixir (Oban, Postgrex.Notifications), Postgres (grid_tasks work queue + LISTEN/NOTIFY), Kubernetes/flux.


Work Streams Overview

Stream Priority What Payoff
A: Port f00 1 Move f00 chain step (native duct + NEXRAD + commercial + ProfilesFile write) from Elixir to Rust. Finishes hourly-chain memory story. Hot pod can drop from 2Gi/replica × N replicas to 1Gi.
B: Delete MRMS 2 Remove MrmsFetchWorker, MrmsClient, MrmsCache, and the */2 * * * * cron. Removes 30 GRIB2 decodes/hour of dead work. Zero risk — AsosAdjustmentWorker (only consumer) is already disabled.
C: Port HrrrFetchWorker 3 Move per-QSO HRRR profile fetch to Rust. New hrrr_point_tasks queue. Elixir enqueues, Rust fulfills. Takes the :hrrr queue off BEAM. Removes per-QSO refc-binary churn and the explicit :erlang.garbage_collect that's papering over it.

Each stream is independently mergeable and reversible. Recommended order: B → A → C. B first because it's a pure delete (simplest; shrinks the attack surface before A touches more code). A second because it's the real memory win. C last because it's optional polish.


Stream B: Delete MRMS (start here — warm-up)

Context: MrmsFetchWorker runs every 2 minutes on the :propagation queue. It decodes an MRMS PrecipRate GRIB2 and caches the regridded result in MrmsCache. The only consumer that reads that cache, AsosAdjustmentWorker, is disabled in all three configs (see config/runtime.exs:212-216). Net effect: 30 GRIB2 decodes/hour of dead work on the hot pods.

Task B1: Confirm no live consumers

Files to check:

  • Search for MrmsCache and Mrms\. across lib/ and test/ — confirm only the worker/client/cache/tests reference it.
  • Search for AsosAdjustmentWorker — confirm the only references are disabled cron entries + the worker file itself.

Step 1: Run the searches

grep -rn "MrmsCache\|Mrms\." lib/ config/ | grep -v "#"
grep -rn "AsosAdjustmentWorker" config/ lib/

Expected output: References only in workers/mrms_fetch_worker.ex, weather/mrms_client.ex, weather/mrms_cache.ex, application.ex (supervisor), and commented-out cron lines. No LiveView, context, or worker depends on it.

If you find a live consumer, STOP — add that consumer to the removal list before proceeding.

Task B2: Remove MRMS from supervisor + cron

Files:

  • Modify: lib/microwaveprop/application.ex (drop MrmsCache from the supervision tree)
  • Modify: config/runtime.exs:206-211 (drop the MrmsFetchWorker cron line and its comment block)
  • Modify: config/config.exs:101 (same removal in dev-carrying config)
  • Modify: config/dev.exs:100 (same)

Step 1: Write the failing test

Add to test/microwaveprop/application_test.exs (create if missing):

defmodule Microwaveprop.ApplicationTest do
  use ExUnit.Case, async: true

  test "MrmsCache is not in the supervision tree" do
    children = Microwaveprop.Application.children_for_env(:test)
    refute Enum.any?(children, &match?({Microwaveprop.Weather.MrmsCache, _}, &1))
    refute Enum.any?(children, fn
      Microwaveprop.Weather.MrmsCache -> true
      _ -> false
    end)
  end
end

If children_for_env/1 doesn't exist, either extract it from start/2 or inline the list in the test.

Step 2: Run test to verify it fails

mix test test/microwaveprop/application_test.exs

Expected: FAIL — MrmsCache still in supervision tree.

Step 3: Remove MrmsCache from supervision tree

Edit lib/microwaveprop/application.ex, delete the line that adds MrmsCache.

Step 4: Run test to verify it passes

mix test test/microwaveprop/application_test.exs

Expected: PASS.

Step 5: Remove cron entries

Delete the MrmsFetchWorker entry + its leading comment block from all three config files.

Step 6: Commit

git add lib/microwaveprop/application.ex config/ test/microwaveprop/application_test.exs
git commit -m "chore(mrms): remove from supervision tree and cron (consumer disabled)"

Task B3: Delete MRMS source files and tests

Files:

  • Delete: lib/microwaveprop/workers/mrms_fetch_worker.ex
  • Delete: lib/microwaveprop/weather/mrms_client.ex
  • Delete: lib/microwaveprop/weather/mrms_cache.ex
  • Delete: test/microwaveprop/workers/mrms_fetch_worker_test.exs
  • Delete: test/microwaveprop/weather/mrms_cache_test.exs
  • Modify: config/test.exs:66 (drop mrms_req_options)

Step 1: Delete files

rm lib/microwaveprop/workers/mrms_fetch_worker.ex \
   lib/microwaveprop/weather/mrms_client.ex \
   lib/microwaveprop/weather/mrms_cache.ex \
   test/microwaveprop/workers/mrms_fetch_worker_test.exs \
   test/microwaveprop/weather/mrms_cache_test.exs

Step 2: Drop the test config line

Edit config/test.exs:66, remove the mrms_req_options line.

Step 3: Run full test suite

mix precommit

Expected: clean compile (warnings-as-errors), all tests pass. If mix xref surfaces a caller, fix it.

Step 4: Commit

git add -A
git commit -m "chore(mrms): delete worker, client, cache, and tests"

Task B4: Drop MRMS k8s resources (if any)

Files to inspect:

  • k8s/kustomization.yaml, any mrms_* resource

Step 1: Check

grep -r "mrms\|Mrms" k8s/

Step 2: If found, remove. Otherwise skip.

Step 3: Commit if changed.


Stream A: Port f00 to Rust

Context: Today, Elixir's PropagationGridWorker.perform/1 handles f00 only — the analysis-hour step that does three things Rust doesn't: native-level HRRR duct merge (the ~530 MB native GRIB2), NEXRAD composite reflectivity overlay, and commercial-link degradation lookup. It also writes the ProfilesFile that /weather reads. Forecast hours f01f18 already run in Rust via the grid_tasks queue.

Strategy: Add a kind column to grid_tasks so the same queue serves both forecast hours (kind='forecast') and analysis-hour enrichment (kind='analysis'). Rust's worker dispatches on kind. Elixir's seeder switches from Oban.insert_all for f00 to seeding a kind='analysis' row alongside the 18 kind='forecast' rows. When the analysis task completes, Rust writes the same /data/scores/<band>/<iso>.ntms + /data/profiles/<iso>.ntms files and NOTIFYs propagation_ready with the existing payload.

Task A1: Add kind column to grid_tasks

Files:

  • Create: priv/repo/migrations/YYYYMMDDHHMMSS_add_kind_to_grid_tasks.exs

Step 1: Generate migration

mix ecto.gen.migration add_kind_to_grid_tasks

Step 2: Write the migration

defmodule Microwaveprop.Repo.Migrations.AddKindToGridTasks do
  use Ecto.Migration

  def change do
    alter table(:grid_tasks) do
      add :kind, :string, null: false, default: "forecast"
    end

    create index(:grid_tasks, [:status, :kind])

    # Replace the old unique index with one that includes kind so an
    # analysis task and a forecast task for the same (run_time, f00)
    # can coexist without conflict.
    drop_if_exists unique_index(:grid_tasks, [:run_time, :forecast_hour])
    create unique_index(:grid_tasks, [:run_time, :forecast_hour, :kind])
  end
end

Step 3: Run migration

mix ecto.migrate

Step 4: Commit

git add priv/repo/migrations/
git commit -m "feat(grid_tasks): add kind column to separate forecast vs analysis work"

Task A2: Rust reads kind and routes

Files:

  • Modify: rust/prop_grid_rs/src/db.rs — add kind: String to the task struct, select it in claim_next
  • Modify: rust/prop_grid_rs/src/bin/worker.rs — match on kind and dispatch to forecast vs. analysis pipeline
  • Test: rust/prop_grid_rs/src/db.rs test for claim_next returning the kind

Step 1: Write failing test

In rust/prop_grid_rs/src/db.rs:

#[tokio::test]
async fn claim_next_returns_kind() {
    // seed one task with kind='analysis', claim it, assert kind round-trips
    // ...
}

Step 2: Run test

cd rust/prop_grid_rs && cargo test claim_next_returns_kind

Expected: FAIL (field doesn't exist).

Step 3: Add kind field to the claim struct + SELECT list

Minimal change to claim_next to read and return kind. Default to "forecast" if unmapped so the compile stays green during rollout.

Step 4: Run test

Expected: PASS.

Step 5: Commit

git add rust/prop_grid_rs/src/db.rs
git commit -m "feat(prop-grid-rs): surface grid_tasks.kind in claim_next"

Task A3: Port HrrrNativeClient.fetch_native_duct_grid to Rust

This is the biggest subtask — the 530 MB GRIB2 native-level pull + duct-metric computation.

Files:

  • Create: rust/prop_grid_rs/src/native_duct.rs
  • Test: rust/prop_grid_rs/src/native_duct.rs (inline #[cfg(test)] mod tests)

Step 1: Write golden-fixture test

Follow the same pattern Phase 1 used for surface/pressure: a Mix task dumps the Elixir-computed duct grid for a known (date, hour) to a bincode file in rust/prop_grid_rs/tests/golden/. Rust loads it and asserts exact match on every {lat, lon} key and per-field values.

Create lib/mix/tasks/rust.golden_duct.ex:

defmodule Mix.Tasks.Rust.GoldenDuct do
  use Mix.Task

  @shortdoc "Dump Elixir-computed native duct grid for Rust comparison"

  def run([date_str, hour_str]) do
    Mix.Task.run("app.start")
    date = Date.from_iso8601!(date_str)
    hour = String.to_integer(hour_str)
    grid_spec = Microwaveprop.Propagation.Grid.wgrib2_grid_spec()

    {:ok, duct_grid} =
      Microwaveprop.Weather.HrrrNativeClient.fetch_native_duct_grid(date, hour, grid_spec, 0)

    path = "rust/prop_grid_rs/tests/golden/duct_#{date_str}_t#{hour_str}z.bincode"
    File.mkdir_p!(Path.dirname(path))
    File.write!(path, :erlang.term_to_binary(duct_grid))
    IO.puts("Wrote #{map_size(duct_grid)} cells to #{path}")
  end
end

Step 2: Run the golden dump against a recent hour

mix rust.golden_duct 2026-04-19 14

Expected: writes a bincode file with ~92k cells.

Step 3: Write the Rust failing test

In rust/prop_grid_rs/src/native_duct.rs:

#[tokio::test]
async fn matches_elixir_golden() {
    let golden = load_golden("tests/golden/duct_2026-04-19_t14z.bincode");
    let rust = fetch_native_duct_grid(Date::from_ymd(2026, 4, 19), 14, grid_spec(), 0).await.unwrap();

    assert_eq!(rust.len(), golden.len());
    for (pt, expected) in &golden {
        let got = rust.get(pt).expect(&format!("missing {:?}", pt));
        assert!((got.native_min_gradient - expected.native_min_gradient).abs() < 1e-3);
        assert_eq!(got.duct_count, expected.duct_count);
        // ... other fields
    }
}

Step 4: Run test

cargo test matches_elixir_golden

Expected: FAIL — fetch_native_duct_grid doesn't exist.

Step 5: Implement the Rust port

Port these pieces in order, one commit each:

  1. Byte-range fetch for duct variables. Port duct_byte_ranges/1 (selects TMP/SPFH/HGT/PRES on all 50 hybrid levels from the idx). Reuse the existing fetcher::download_ranges_to_file that Phase 1 built.
  2. wgrib2 subprocess for extract_grid_from_file_mapped. Port the -lola + match-pattern invocation. Parse the flat output into a HashMap<(lat, lon), HashMap<String, f32>>. Stream cell-by-cell rather than buffering all values — matches Elixir's "peak memory ~86 MB" design.
  3. compute_duct_metrics per cell. Port build_native_profile/1, min_m_gradient/1, and Duct.analyze/1. Duct.analyze/1 is in lib/microwaveprop/propagation/duct.ex — that's its own port.
  4. Wire into native_duct.rs::fetch_native_duct_grid as the public entry point.

Step 6: Run test to verify it passes

cargo test matches_elixir_golden

Expected: PASS with max-per-cell delta under 1e-3.

Step 7: Commit

git add rust/prop_grid_rs/src/native_duct.rs rust/prop_grid_rs/tests/golden/ lib/mix/tasks/rust.golden_duct.ex
git commit -m "feat(prop-grid-rs): port HrrrNativeClient.fetch_native_duct_grid"

Task A4: Port NEXRAD composite reflectivity fetch

Context: NexradClient.fetch_frame/2 (in lib/microwaveprop/weather/nexrad_client.ex) pulls the current MRMS composite reflectivity mosaic and returns [%{lat, lon, max_reflectivity_dbz}]. The merge into grid_data is line-level (apply_nexrad_observations). It's another GRIB2 decode but much smaller than the native-level file.

Files:

  • Create: rust/prop_grid_rs/src/nexrad.rs
  • Test: inline fixture test

Step 1: Golden fixture

Same pattern: mix rust.golden_nexrad 2026-04-19T14:00:00Z dumps the Elixir observations list as bincode.

Step 2: Write failing test

#[tokio::test]
async fn matches_elixir_golden() { /* ... */ }

Step 3: Implement fetch_frame by porting the HTTP GET + GRIB2 decode from NexradClient.

Step 4: Run test. Expected PASS.

Step 5: Commit.

Context: Commercial.build_link_lookup/1 (in lib/microwaveprop/commercial.ex) returns a [{{lat, lon}, degradation}] list from Postgres. link_degradation_from_lookup/2 is a pure haversine match. Only 7 links cluster around DFW so most cells return nil.

Files:

  • Create: rust/prop_grid_rs/src/commercial.rs
  • Test: inline SQL + haversine round-trip

Step 1: Golden fixture (simpler — just dump the 7 link rows)

Step 2: Failing test

#[tokio::test]
async fn haversine_matches_elixir() { /* 7 links × 10 sample points */ }

Step 3: Port build_link_lookup (one SELECT) + haversine matcher.

Step 4: Commit.

Task A6: Port ProfilesFile write

Context: Microwaveprop.Propagation.ProfilesFile.write!/2 serializes the fully-enriched grid_data to /data/profiles/<iso>.ntms using :erlang.term_to_binary/1. This is the tricky one — the file format is Erlang ETF.

Decision point: Two options.

  1. Port the reader to Rust + ETF encoder. The eetf crate exists but is not heavily maintained. Risk: ETF subtleties (ref counting, atom tables) could produce bit-different output that breaks Elixir's reader.
  2. Change the format. Replace ETF with a simple framed bincode/cbor/msgpack file. Rust writes, Elixir reads via a new decoder. Clean break.

Recommendation: Option 2. The profiles file is read from exactly one place (/weather and point-detail LiveView in lib/microwaveprop/weather.ex). Porting the reader to a new format is ~50 lines. Fighting with eetf is a tarpit.

Files:

  • Create: rust/prop_grid_rs/src/profiles_file.rs (writes new format)
  • Modify: lib/microwaveprop/propagation/profiles_file.ex (reads old format; add reader for new format; write-path deleted once Rust owns f00)
  • Modify: lib/microwaveprop/weather.ex (call sites that read profiles)

Step 1: Design the format. Header: magic PROF\x00, version u8, record count u32, {lat: f32, lon: f32, field_count: u8, [{key: varint-len-prefixed string, value: tagged-union}]...}. Or — simpler — MessagePack. rmp-serde in Rust, Msgpax in Elixir. Both are well-maintained.

Step 2: Write Elixir reader for MessagePack variant. Keep the old ETF reader around under a read_etf!/1 alias for backward compatibility during rollout.

Step 3: Failing test in Rust. Round-trip: write a sample grid_data, read it back, assert equal.

Step 4: Implement writer.

Step 5: Dual-read test in Elixir. Old ETF files still read; new MessagePack files read identically.

Step 6: Commit.

Task A7: Wire analysis pipeline in Rust worker

Files:

  • Modify: rust/prop_grid_rs/src/pipeline.rs — add run_analysis_task alongside the existing forecast pipeline
  • Modify: rust/prop_grid_rs/src/bin/worker.rs — dispatch on task.kind

Step 1: Write failing integration test

In rust/prop_grid_rs/tests/analysis_pipeline_test.rs:

#[tokio::test]
async fn analysis_task_writes_scores_and_profiles() {
    // seed one kind='analysis' task, run pipeline, assert
    //   /data/scores/10000/<iso>.ntms exists
    //   /data/profiles/<iso>.ntms exists
    //   NOTIFY propagation_ready fired
}

Step 2: Run test. Expected FAIL.

Step 3: Implement run_analysis_task. Same shape as forecast pipeline but:

  • Calls native_duct::fetch_native_duct_grid after the regular surface+pressure merge.
  • Calls nexrad::fetch_frame and merges.
  • Calls commercial::build_link_lookup and merges.
  • Writes ProfilesFile to /data/profiles/<iso>.ntms.
  • Scores all bands as before.

Step 4: Run test. Expected PASS.

Step 5: Commit.

Task A8: Flip Elixir seeder to enqueue analysis as grid_task

Files:

  • Modify: lib/microwaveprop/workers/propagation_grid_worker.ex:80-99 (seed_chain)
  • Modify: lib/microwaveprop/propagation/grid_task_enqueuer.ex (add seed_with_analysis/1)

Step 1: Write failing test

In test/microwaveprop/workers/propagation_grid_worker_test.exs, update the seed test to assert:

  • Zero Oban jobs enqueued (f00 is no longer an Oban job).
  • 19 grid_tasks rows: 1 with kind='analysis' AND forecast_hour=0, 18 with kind='forecast' AND forecast_hour IN (1..18).

Step 2: Run test. Expected FAIL.

Step 3: Modify seed_chain to call GridTaskEnqueuer.seed_with_analysis(run_time) and return :ok. Delete the Oban.insert_all line. Delete run_chain_step, process_forecast_hour, and all the merge helpers from propagation_grid_worker.ex. The module becomes a thin cron handler.

Step 4: Run test. Expected PASS.

Step 5: Delete now-unused Elixir modules after mix xref callers confirms no callers remain:

  • lib/microwaveprop/weather/hrrr_native_client.ex
  • lib/microwaveprop/weather/nexrad_client.ex (assuming no other caller)
  • The merge_native_duct_data, merge_nexrad_data, merge_commercial_link_data, compute_scores_* helpers

Step 6: Commit.

Task A9: Shrink hot pod memory limit

Files:

  • Modify: k8s/deployment.yaml

Step 1: Wait for 24h of clean post-cutover runs.

Step 2: Reduce memory limit from 2Gi to 1Gi, requests from 512Mi to 256Mi.

Step 3: Watch kubectl -n prop top pod for OOM signals over 24h. If any pod OOMKills, revert immediately and investigate before retrying.

Step 4: Commit.


Stream C: Port HrrrFetchWorker (optional polish)

Context: Each QSO submission enqueues an HrrrFetchWorker job that fetches a HRRR profile at the contact location. Per-job payload is small (one point, ~100 kB of GRIB2 bytes) but the job calls HrrrClient which on BEAM holds refc binaries that the worker explicitly :erlang.garbage_collects at the end. That GC is a signal the binary pressure is real.

Strategy: New Rust binary hrrr-point-worker shares the existing prop_grid_rs crate (adds a second bin). A new hrrr_point_tasks table mirrors grid_tasks. Elixir's ingestion switches from HrrrFetchWorker (Oban) to inserting into hrrr_point_tasks.

Task C1: Migration for hrrr_point_tasks

Files:

  • Create: priv/repo/migrations/YYYYMMDDHHMMSS_create_hrrr_point_tasks.exs

Columns: id uuid PK, qso_id uuid, valid_time timestamp, lat double precision, lon double precision, status text, claimed_at, completed_at, error text, attempt smallint. Unique on (qso_id, valid_time).

Step 1: Generate + write migration. Step 2: Run migration. Step 3: Commit.

Task C2: Rust hrrr-point-worker binary

Files:

  • Create: rust/prop_grid_rs/src/bin/hrrr_point_worker.rs
  • Create: rust/prop_grid_rs/src/point_fetcher.rs (per-point HRRR profile extract)

Step 1: Golden fixture for HrrrClient.fetch_profile. Step 2: Failing test. Step 3: Implement point fetcher + worker loop. Reuse the existing fetcher module for byte-range downloads; add a new extract_point_profile that takes a single (lat, lon) instead of a grid spec. Step 4: Commit.

Task C3: k8s deployment for hrrr-point-worker

Files:

  • Create: k8s/deployment-hrrr-point-rs.yaml (same image, different ENTRYPOINT)
  • Modify: k8s/kustomization.yaml

Step 1: Write manifest. 1 replica on talos5 initially, 256 Mi memory. Step 2: Apply. Step 3: Commit.

Task C4: Switch Elixir ingestion to enqueue into hrrr_point_tasks

Files:

  • Modify: wherever HrrrFetchWorker.new(...) is called (check lib/microwaveprop/qsos.ex or lib/microwaveprop/radio.ex via grep -rn "HrrrFetchWorker.new")
  • Delete: lib/microwaveprop/workers/hrrr_fetch_worker.ex (after mix xref callers is clean)
  • Delete: corresponding test

Step 1: Failing test. QSO submission inserts a row into hrrr_point_tasks and does NOT enqueue an Oban job. Step 2: Implement. Step 3: Commit.


Critical Files Reference

Ported out (deleted at end of Stream A):

  • lib/microwaveprop/weather/hrrr_native_client.ex
  • lib/microwaveprop/weather/nexrad_client.ex (confirm no per-QSO callers first)
  • lib/microwaveprop/weather/mrms_client.ex (Stream B)
  • lib/microwaveprop/weather/mrms_cache.ex (Stream B)
  • lib/microwaveprop/workers/mrms_fetch_worker.ex (Stream B)
  • lib/microwaveprop/workers/hrrr_fetch_worker.ex (Stream C)
  • Most of lib/microwaveprop/workers/propagation_grid_worker.ex — becomes a thin seeder

Kept as-is:

  • lib/microwaveprop/commercial.ex — Elixir still needs this for the SNMP poller and LiveView sensor widget. Rust just adds a parallel reader over the same commercial_samples table.
  • lib/microwaveprop/propagation/duct.ex — ported to Rust, original stays because other callers (per-contact analysis) still use it.

New Rust modules (Stream A):

  • rust/prop_grid_rs/src/native_duct.rs
  • rust/prop_grid_rs/src/nexrad.rs
  • rust/prop_grid_rs/src/commercial.rs
  • rust/prop_grid_rs/src/profiles_file.rs
  • rust/prop_grid_rs/src/duct.rs (ported from lib/microwaveprop/propagation/duct.ex)

New Rust module (Stream C):

  • rust/prop_grid_rs/src/point_fetcher.rs
  • rust/prop_grid_rs/src/bin/hrrr_point_worker.rs

Verification

Stream B:

  • grep -r "Mrms" lib/ test/ config/ returns nothing.
  • mix precommit green.
  • 24h later: kubectl -n prop get events shows no MrmsFetchWorker executions.

Stream A:

  • cargo test -p prop_grid_rs green with golden-duct, golden-nexrad, golden-commercial fixtures.
  • mix test green.
  • One full post-cutover hourly cycle shows kind='analysis' task status='done' in grid_tasks, /data/profiles/<iso>.ntms written, /weather page loads correctly.
  • Hot pod beam_memory_total < 700 MiB peak over 24 h (down from ~1 GiB under Phase 2).
  • Hot pod memory limit reduced to 1 Gi with zero OOMKilled events over 7 days.

Stream C:

  • QSO submission inserts a hrrr_point_tasks row within 1 s.
  • Rust hrrr-point-worker logs show claim→complete within 30 s of insert.
  • HRRR profile appears on the QSO detail page within 1 min.
  • :hrrr Oban queue is empty and can be removed from the Oban config.

Rollback Plans

Stream B: git revert restores MRMS. Supervision tree re-adds MrmsCache on next deploy. No data loss — MrmsCache was in-memory only.

Stream A: git revert restores the Elixir f00 chain step. The grid_tasks.kind column can stay (default 'forecast' keeps existing Rust behaviour). Bump hot pod memory back to 2Gi if Phase 3 shrink was already deployed.

Stream C: git revert restores HrrrFetchWorker. hrrr_point_tasks rows become orphan; a one-off Mix task migrates them back to Oban jobs.