MrmsFetchWorker fired every 2 min on the :propagation queue, decoding an MRMS PrecipRate GRIB2 into MrmsCache. Its only consumer, AsosAdjustmentWorker, is disabled in all configs (dev / config / runtime). Net effect: 30 GRIB2 decodes/hour of dead work on hot pods. Removed: - MrmsFetchWorker, MrmsClient, MrmsCache and their tests - cron entries in config.exs, dev.exs, runtime.exs - mrms_req_options stub in test.exs - MrmsCache from supervision tree in application.ex - stale MRMS references in PropagationGridWorker docstring and test Also adds docs/plans/2026-04-19-rust-migration-phase3.md which tracks the broader Stream A (f00 port) and Stream C (HrrrFetchWorker port) follow-on work.
24 KiB
Rust Migration Phase 3 — f00 Port + MRMS Deletion + HRRR Per-QSO Port
For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
Goal: Remove every remaining GRIB2 decode from BEAM so hot pods can shrink to 1 Gi, and delete one dead pipeline (MRMS) along the way.
Architecture: Three independent work streams against the same shared prop-grid-rs binary. Stream A ports f00's native-duct + NEXRAD + commercial-link merge from Elixir into Rust via a new grid_tasks.kind column so the same work queue handles both forecast hours and analysis-hour enrichment. Stream B deletes the MRMS worker/cache/client entirely (the only consumer AsosAdjustmentWorker is already disabled). Stream C ports the per-QSO HrrrFetchWorker into a new Rust binary sharing the same crate, taking the :hrrr queue off BEAM.
Tech Stack: Rust (tokio, sqlx, reqwest, wgrib2 subprocess), Elixir (Oban, Postgrex.Notifications), Postgres (grid_tasks work queue + LISTEN/NOTIFY), Kubernetes/flux.
Work Streams Overview
| Stream | Priority | What | Payoff |
|---|---|---|---|
| A: Port f00 | 1 | Move f00 chain step (native duct + NEXRAD + commercial + ProfilesFile write) from Elixir to Rust. | Finishes hourly-chain memory story. Hot pod can drop from 2Gi/replica × N replicas to 1Gi. |
| B: Delete MRMS | 2 | Remove MrmsFetchWorker, MrmsClient, MrmsCache, and the */2 * * * * cron. |
Removes 30 GRIB2 decodes/hour of dead work. Zero risk — AsosAdjustmentWorker (only consumer) is already disabled. |
| C: Port HrrrFetchWorker | 3 | Move per-QSO HRRR profile fetch to Rust. New hrrr_point_tasks queue. Elixir enqueues, Rust fulfills. |
Takes the :hrrr queue off BEAM. Removes per-QSO refc-binary churn and the explicit :erlang.garbage_collect that's papering over it. |
Each stream is independently mergeable and reversible. Recommended order: B → A → C. B first because it's a pure delete (simplest; shrinks the attack surface before A touches more code). A second because it's the real memory win. C last because it's optional polish.
Stream B: Delete MRMS (start here — warm-up)
Context: MrmsFetchWorker runs every 2 minutes on the :propagation queue. It decodes an MRMS PrecipRate GRIB2 and caches the regridded result in MrmsCache. The only consumer that reads that cache, AsosAdjustmentWorker, is disabled in all three configs (see config/runtime.exs:212-216). Net effect: 30 GRIB2 decodes/hour of dead work on the hot pods.
Task B1: Confirm no live consumers
Files to check:
- Search for
MrmsCacheandMrms\.acrosslib/andtest/— confirm only the worker/client/cache/tests reference it. - Search for
AsosAdjustmentWorker— confirm the only references are disabled cron entries + the worker file itself.
Step 1: Run the searches
grep -rn "MrmsCache\|Mrms\." lib/ config/ | grep -v "#"
grep -rn "AsosAdjustmentWorker" config/ lib/
Expected output: References only in workers/mrms_fetch_worker.ex, weather/mrms_client.ex, weather/mrms_cache.ex, application.ex (supervisor), and commented-out cron lines. No LiveView, context, or worker depends on it.
If you find a live consumer, STOP — add that consumer to the removal list before proceeding.
Task B2: Remove MRMS from supervisor + cron
Files:
- Modify:
lib/microwaveprop/application.ex(dropMrmsCachefrom the supervision tree) - Modify:
config/runtime.exs:206-211(drop theMrmsFetchWorkercron line and its comment block) - Modify:
config/config.exs:101(same removal in dev-carrying config) - Modify:
config/dev.exs:100(same)
Step 1: Write the failing test
Add to test/microwaveprop/application_test.exs (create if missing):
defmodule Microwaveprop.ApplicationTest do
use ExUnit.Case, async: true
test "MrmsCache is not in the supervision tree" do
children = Microwaveprop.Application.children_for_env(:test)
refute Enum.any?(children, &match?({Microwaveprop.Weather.MrmsCache, _}, &1))
refute Enum.any?(children, fn
Microwaveprop.Weather.MrmsCache -> true
_ -> false
end)
end
end
If children_for_env/1 doesn't exist, either extract it from start/2 or inline the list in the test.
Step 2: Run test to verify it fails
mix test test/microwaveprop/application_test.exs
Expected: FAIL — MrmsCache still in supervision tree.
Step 3: Remove MrmsCache from supervision tree
Edit lib/microwaveprop/application.ex, delete the line that adds MrmsCache.
Step 4: Run test to verify it passes
mix test test/microwaveprop/application_test.exs
Expected: PASS.
Step 5: Remove cron entries
Delete the MrmsFetchWorker entry + its leading comment block from all three config files.
Step 6: Commit
git add lib/microwaveprop/application.ex config/ test/microwaveprop/application_test.exs
git commit -m "chore(mrms): remove from supervision tree and cron (consumer disabled)"
Task B3: Delete MRMS source files and tests
Files:
- Delete:
lib/microwaveprop/workers/mrms_fetch_worker.ex - Delete:
lib/microwaveprop/weather/mrms_client.ex - Delete:
lib/microwaveprop/weather/mrms_cache.ex - Delete:
test/microwaveprop/workers/mrms_fetch_worker_test.exs - Delete:
test/microwaveprop/weather/mrms_cache_test.exs - Modify:
config/test.exs:66(dropmrms_req_options)
Step 1: Delete files
rm lib/microwaveprop/workers/mrms_fetch_worker.ex \
lib/microwaveprop/weather/mrms_client.ex \
lib/microwaveprop/weather/mrms_cache.ex \
test/microwaveprop/workers/mrms_fetch_worker_test.exs \
test/microwaveprop/weather/mrms_cache_test.exs
Step 2: Drop the test config line
Edit config/test.exs:66, remove the mrms_req_options line.
Step 3: Run full test suite
mix precommit
Expected: clean compile (warnings-as-errors), all tests pass. If mix xref surfaces a caller, fix it.
Step 4: Commit
git add -A
git commit -m "chore(mrms): delete worker, client, cache, and tests"
Task B4: Drop MRMS k8s resources (if any)
Files to inspect:
k8s/kustomization.yaml, anymrms_*resource
Step 1: Check
grep -r "mrms\|Mrms" k8s/
Step 2: If found, remove. Otherwise skip.
Step 3: Commit if changed.
Stream A: Port f00 to Rust
Context: Today, Elixir's PropagationGridWorker.perform/1 handles f00 only — the analysis-hour step that does three things Rust doesn't: native-level HRRR duct merge (the ~530 MB native GRIB2), NEXRAD composite reflectivity overlay, and commercial-link degradation lookup. It also writes the ProfilesFile that /weather reads. Forecast hours f01–f18 already run in Rust via the grid_tasks queue.
Strategy: Add a kind column to grid_tasks so the same queue serves both forecast hours (kind='forecast') and analysis-hour enrichment (kind='analysis'). Rust's worker dispatches on kind. Elixir's seeder switches from Oban.insert_all for f00 to seeding a kind='analysis' row alongside the 18 kind='forecast' rows. When the analysis task completes, Rust writes the same /data/scores/<band>/<iso>.ntms + /data/profiles/<iso>.ntms files and NOTIFYs propagation_ready with the existing payload.
Task A1: Add kind column to grid_tasks
Files:
- Create:
priv/repo/migrations/YYYYMMDDHHMMSS_add_kind_to_grid_tasks.exs
Step 1: Generate migration
mix ecto.gen.migration add_kind_to_grid_tasks
Step 2: Write the migration
defmodule Microwaveprop.Repo.Migrations.AddKindToGridTasks do
use Ecto.Migration
def change do
alter table(:grid_tasks) do
add :kind, :string, null: false, default: "forecast"
end
create index(:grid_tasks, [:status, :kind])
# Replace the old unique index with one that includes kind so an
# analysis task and a forecast task for the same (run_time, f00)
# can coexist without conflict.
drop_if_exists unique_index(:grid_tasks, [:run_time, :forecast_hour])
create unique_index(:grid_tasks, [:run_time, :forecast_hour, :kind])
end
end
Step 3: Run migration
mix ecto.migrate
Step 4: Commit
git add priv/repo/migrations/
git commit -m "feat(grid_tasks): add kind column to separate forecast vs analysis work"
Task A2: Rust reads kind and routes
Files:
- Modify:
rust/prop_grid_rs/src/db.rs— addkind: Stringto the task struct, select it inclaim_next - Modify:
rust/prop_grid_rs/src/bin/worker.rs— match onkindand dispatch to forecast vs. analysis pipeline - Test:
rust/prop_grid_rs/src/db.rstest forclaim_nextreturning thekind
Step 1: Write failing test
In rust/prop_grid_rs/src/db.rs:
#[tokio::test]
async fn claim_next_returns_kind() {
// seed one task with kind='analysis', claim it, assert kind round-trips
// ...
}
Step 2: Run test
cd rust/prop_grid_rs && cargo test claim_next_returns_kind
Expected: FAIL (field doesn't exist).
Step 3: Add kind field to the claim struct + SELECT list
Minimal change to claim_next to read and return kind. Default to "forecast" if unmapped so the compile stays green during rollout.
Step 4: Run test
Expected: PASS.
Step 5: Commit
git add rust/prop_grid_rs/src/db.rs
git commit -m "feat(prop-grid-rs): surface grid_tasks.kind in claim_next"
Task A3: Port HrrrNativeClient.fetch_native_duct_grid to Rust
This is the biggest subtask — the 530 MB GRIB2 native-level pull + duct-metric computation.
Files:
- Create:
rust/prop_grid_rs/src/native_duct.rs - Test:
rust/prop_grid_rs/src/native_duct.rs(inline#[cfg(test)] mod tests)
Step 1: Write golden-fixture test
Follow the same pattern Phase 1 used for surface/pressure: a Mix task dumps the Elixir-computed duct grid for a known (date, hour) to a bincode file in rust/prop_grid_rs/tests/golden/. Rust loads it and asserts exact match on every {lat, lon} key and per-field values.
Create lib/mix/tasks/rust.golden_duct.ex:
defmodule Mix.Tasks.Rust.GoldenDuct do
use Mix.Task
@shortdoc "Dump Elixir-computed native duct grid for Rust comparison"
def run([date_str, hour_str]) do
Mix.Task.run("app.start")
date = Date.from_iso8601!(date_str)
hour = String.to_integer(hour_str)
grid_spec = Microwaveprop.Propagation.Grid.wgrib2_grid_spec()
{:ok, duct_grid} =
Microwaveprop.Weather.HrrrNativeClient.fetch_native_duct_grid(date, hour, grid_spec, 0)
path = "rust/prop_grid_rs/tests/golden/duct_#{date_str}_t#{hour_str}z.bincode"
File.mkdir_p!(Path.dirname(path))
File.write!(path, :erlang.term_to_binary(duct_grid))
IO.puts("Wrote #{map_size(duct_grid)} cells to #{path}")
end
end
Step 2: Run the golden dump against a recent hour
mix rust.golden_duct 2026-04-19 14
Expected: writes a bincode file with ~92k cells.
Step 3: Write the Rust failing test
In rust/prop_grid_rs/src/native_duct.rs:
#[tokio::test]
async fn matches_elixir_golden() {
let golden = load_golden("tests/golden/duct_2026-04-19_t14z.bincode");
let rust = fetch_native_duct_grid(Date::from_ymd(2026, 4, 19), 14, grid_spec(), 0).await.unwrap();
assert_eq!(rust.len(), golden.len());
for (pt, expected) in &golden {
let got = rust.get(pt).expect(&format!("missing {:?}", pt));
assert!((got.native_min_gradient - expected.native_min_gradient).abs() < 1e-3);
assert_eq!(got.duct_count, expected.duct_count);
// ... other fields
}
}
Step 4: Run test
cargo test matches_elixir_golden
Expected: FAIL — fetch_native_duct_grid doesn't exist.
Step 5: Implement the Rust port
Port these pieces in order, one commit each:
- Byte-range fetch for duct variables. Port
duct_byte_ranges/1(selects TMP/SPFH/HGT/PRES on all 50 hybrid levels from the idx). Reuse the existingfetcher::download_ranges_to_filethat Phase 1 built. - wgrib2 subprocess for
extract_grid_from_file_mapped. Port the-lola+ match-pattern invocation. Parse the flat output into aHashMap<(lat, lon), HashMap<String, f32>>. Stream cell-by-cell rather than buffering all values — matches Elixir's "peak memory ~86 MB" design. compute_duct_metricsper cell. Portbuild_native_profile/1,min_m_gradient/1, andDuct.analyze/1.Duct.analyze/1is inlib/microwaveprop/propagation/duct.ex— that's its own port.- Wire into
native_duct.rs::fetch_native_duct_gridas the public entry point.
Step 6: Run test to verify it passes
cargo test matches_elixir_golden
Expected: PASS with max-per-cell delta under 1e-3.
Step 7: Commit
git add rust/prop_grid_rs/src/native_duct.rs rust/prop_grid_rs/tests/golden/ lib/mix/tasks/rust.golden_duct.ex
git commit -m "feat(prop-grid-rs): port HrrrNativeClient.fetch_native_duct_grid"
Task A4: Port NEXRAD composite reflectivity fetch
Context: NexradClient.fetch_frame/2 (in lib/microwaveprop/weather/nexrad_client.ex) pulls the current MRMS composite reflectivity mosaic and returns [%{lat, lon, max_reflectivity_dbz}]. The merge into grid_data is line-level (apply_nexrad_observations). It's another GRIB2 decode but much smaller than the native-level file.
Files:
- Create:
rust/prop_grid_rs/src/nexrad.rs - Test: inline fixture test
Step 1: Golden fixture
Same pattern: mix rust.golden_nexrad 2026-04-19T14:00:00Z dumps the Elixir observations list as bincode.
Step 2: Write failing test
#[tokio::test]
async fn matches_elixir_golden() { /* ... */ }
Step 3: Implement fetch_frame by porting the HTTP GET + GRIB2 decode from NexradClient.
Step 4: Run test. Expected PASS.
Step 5: Commit.
Task A5: Port commercial-link degradation lookup
Context: Commercial.build_link_lookup/1 (in lib/microwaveprop/commercial.ex) returns a [{{lat, lon}, degradation}] list from Postgres. link_degradation_from_lookup/2 is a pure haversine match. Only 7 links cluster around DFW so most cells return nil.
Files:
- Create:
rust/prop_grid_rs/src/commercial.rs - Test: inline SQL + haversine round-trip
Step 1: Golden fixture (simpler — just dump the 7 link rows)
Step 2: Failing test
#[tokio::test]
async fn haversine_matches_elixir() { /* 7 links × 10 sample points */ }
Step 3: Port build_link_lookup (one SELECT) + haversine matcher.
Step 4: Commit.
Task A6: Port ProfilesFile write
Context: Microwaveprop.Propagation.ProfilesFile.write!/2 serializes the fully-enriched grid_data to /data/profiles/<iso>.ntms using :erlang.term_to_binary/1. This is the tricky one — the file format is Erlang ETF.
Decision point: Two options.
- Port the reader to Rust + ETF encoder. The
eetfcrate exists but is not heavily maintained. Risk: ETF subtleties (ref counting, atom tables) could produce bit-different output that breaks Elixir's reader. - Change the format. Replace ETF with a simple framed bincode/cbor/msgpack file. Rust writes, Elixir reads via a new decoder. Clean break.
Recommendation: Option 2. The profiles file is read from exactly one place (/weather and point-detail LiveView in lib/microwaveprop/weather.ex). Porting the reader to a new format is ~50 lines. Fighting with eetf is a tarpit.
Files:
- Create:
rust/prop_grid_rs/src/profiles_file.rs(writes new format) - Modify:
lib/microwaveprop/propagation/profiles_file.ex(reads old format; add reader for new format; write-path deleted once Rust owns f00) - Modify:
lib/microwaveprop/weather.ex(call sites that read profiles)
Step 1: Design the format. Header: magic PROF\x00, version u8, record count u32, {lat: f32, lon: f32, field_count: u8, [{key: varint-len-prefixed string, value: tagged-union}]...}. Or — simpler — MessagePack. rmp-serde in Rust, Msgpax in Elixir. Both are well-maintained.
Step 2: Write Elixir reader for MessagePack variant. Keep the old ETF reader around under a read_etf!/1 alias for backward compatibility during rollout.
Step 3: Failing test in Rust. Round-trip: write a sample grid_data, read it back, assert equal.
Step 4: Implement writer.
Step 5: Dual-read test in Elixir. Old ETF files still read; new MessagePack files read identically.
Step 6: Commit.
Task A7: Wire analysis pipeline in Rust worker
Files:
- Modify:
rust/prop_grid_rs/src/pipeline.rs— addrun_analysis_taskalongside the existing forecast pipeline - Modify:
rust/prop_grid_rs/src/bin/worker.rs— dispatch ontask.kind
Step 1: Write failing integration test
In rust/prop_grid_rs/tests/analysis_pipeline_test.rs:
#[tokio::test]
async fn analysis_task_writes_scores_and_profiles() {
// seed one kind='analysis' task, run pipeline, assert
// /data/scores/10000/<iso>.ntms exists
// /data/profiles/<iso>.ntms exists
// NOTIFY propagation_ready fired
}
Step 2: Run test. Expected FAIL.
Step 3: Implement run_analysis_task. Same shape as forecast pipeline but:
- Calls
native_duct::fetch_native_duct_gridafter the regular surface+pressure merge. - Calls
nexrad::fetch_frameand merges. - Calls
commercial::build_link_lookupand merges. - Writes
ProfilesFileto/data/profiles/<iso>.ntms. - Scores all bands as before.
Step 4: Run test. Expected PASS.
Step 5: Commit.
Task A8: Flip Elixir seeder to enqueue analysis as grid_task
Files:
- Modify:
lib/microwaveprop/workers/propagation_grid_worker.ex:80-99(seed_chain) - Modify:
lib/microwaveprop/propagation/grid_task_enqueuer.ex(addseed_with_analysis/1)
Step 1: Write failing test
In test/microwaveprop/workers/propagation_grid_worker_test.exs, update the seed test to assert:
- Zero Oban jobs enqueued (f00 is no longer an Oban job).
- 19 grid_tasks rows: 1 with
kind='analysis' AND forecast_hour=0, 18 withkind='forecast' AND forecast_hour IN (1..18).
Step 2: Run test. Expected FAIL.
Step 3: Modify seed_chain to call GridTaskEnqueuer.seed_with_analysis(run_time) and return :ok. Delete the Oban.insert_all line. Delete run_chain_step, process_forecast_hour, and all the merge helpers from propagation_grid_worker.ex. The module becomes a thin cron handler.
Step 4: Run test. Expected PASS.
Step 5: Delete now-unused Elixir modules after mix xref callers confirms no callers remain:
lib/microwaveprop/weather/hrrr_native_client.exlib/microwaveprop/weather/nexrad_client.ex(assuming no other caller)- The
merge_native_duct_data,merge_nexrad_data,merge_commercial_link_data,compute_scores_*helpers
Step 6: Commit.
Task A9: Shrink hot pod memory limit
Files:
- Modify:
k8s/deployment.yaml
Step 1: Wait for 24h of clean post-cutover runs.
Step 2: Reduce memory limit from 2Gi to 1Gi, requests from 512Mi to 256Mi.
Step 3: Watch kubectl -n prop top pod for OOM signals over 24h. If any pod OOMKills, revert immediately and investigate before retrying.
Step 4: Commit.
Stream C: Port HrrrFetchWorker (optional polish)
Context: Each QSO submission enqueues an HrrrFetchWorker job that fetches a HRRR profile at the contact location. Per-job payload is small (one point, ~100 kB of GRIB2 bytes) but the job calls HrrrClient which on BEAM holds refc binaries that the worker explicitly :erlang.garbage_collects at the end. That GC is a signal the binary pressure is real.
Strategy: New Rust binary hrrr-point-worker shares the existing prop_grid_rs crate (adds a second bin). A new hrrr_point_tasks table mirrors grid_tasks. Elixir's ingestion switches from HrrrFetchWorker (Oban) to inserting into hrrr_point_tasks.
Task C1: Migration for hrrr_point_tasks
Files:
- Create:
priv/repo/migrations/YYYYMMDDHHMMSS_create_hrrr_point_tasks.exs
Columns: id uuid PK, qso_id uuid, valid_time timestamp, lat double precision, lon double precision, status text, claimed_at, completed_at, error text, attempt smallint. Unique on (qso_id, valid_time).
Step 1: Generate + write migration. Step 2: Run migration. Step 3: Commit.
Task C2: Rust hrrr-point-worker binary
Files:
- Create:
rust/prop_grid_rs/src/bin/hrrr_point_worker.rs - Create:
rust/prop_grid_rs/src/point_fetcher.rs(per-point HRRR profile extract)
Step 1: Golden fixture for HrrrClient.fetch_profile.
Step 2: Failing test.
Step 3: Implement point fetcher + worker loop. Reuse the existing fetcher module for byte-range downloads; add a new extract_point_profile that takes a single (lat, lon) instead of a grid spec.
Step 4: Commit.
Task C3: k8s deployment for hrrr-point-worker
Files:
- Create:
k8s/deployment-hrrr-point-rs.yaml(same image, different ENTRYPOINT) - Modify:
k8s/kustomization.yaml
Step 1: Write manifest. 1 replica on talos5 initially, 256 Mi memory. Step 2: Apply. Step 3: Commit.
Task C4: Switch Elixir ingestion to enqueue into hrrr_point_tasks
Files:
- Modify: wherever
HrrrFetchWorker.new(...)is called (checklib/microwaveprop/qsos.exorlib/microwaveprop/radio.exviagrep -rn "HrrrFetchWorker.new") - Delete:
lib/microwaveprop/workers/hrrr_fetch_worker.ex(aftermix xref callersis clean) - Delete: corresponding test
Step 1: Failing test. QSO submission inserts a row into hrrr_point_tasks and does NOT enqueue an Oban job.
Step 2: Implement.
Step 3: Commit.
Critical Files Reference
Ported out (deleted at end of Stream A):
lib/microwaveprop/weather/hrrr_native_client.exlib/microwaveprop/weather/nexrad_client.ex(confirm no per-QSO callers first)lib/microwaveprop/weather/mrms_client.ex(Stream B)lib/microwaveprop/weather/mrms_cache.ex(Stream B)lib/microwaveprop/workers/mrms_fetch_worker.ex(Stream B)lib/microwaveprop/workers/hrrr_fetch_worker.ex(Stream C)- Most of
lib/microwaveprop/workers/propagation_grid_worker.ex— becomes a thin seeder
Kept as-is:
lib/microwaveprop/commercial.ex— Elixir still needs this for the SNMP poller and LiveView sensor widget. Rust just adds a parallel reader over the samecommercial_samplestable.lib/microwaveprop/propagation/duct.ex— ported to Rust, original stays because other callers (per-contact analysis) still use it.
New Rust modules (Stream A):
rust/prop_grid_rs/src/native_duct.rsrust/prop_grid_rs/src/nexrad.rsrust/prop_grid_rs/src/commercial.rsrust/prop_grid_rs/src/profiles_file.rsrust/prop_grid_rs/src/duct.rs(ported fromlib/microwaveprop/propagation/duct.ex)
New Rust module (Stream C):
rust/prop_grid_rs/src/point_fetcher.rsrust/prop_grid_rs/src/bin/hrrr_point_worker.rs
Verification
Stream B:
grep -r "Mrms" lib/ test/ config/returns nothing.mix precommitgreen.- 24h later:
kubectl -n prop get eventsshows no MrmsFetchWorker executions.
Stream A:
cargo test -p prop_grid_rsgreen with golden-duct, golden-nexrad, golden-commercial fixtures.mix testgreen.- One full post-cutover hourly cycle shows
kind='analysis'taskstatus='done'ingrid_tasks,/data/profiles/<iso>.ntmswritten,/weatherpage loads correctly. - Hot pod
beam_memory_total< 700 MiB peak over 24 h (down from ~1 GiB under Phase 2). - Hot pod memory limit reduced to 1 Gi with zero OOMKilled events over 7 days.
Stream C:
- QSO submission inserts a
hrrr_point_tasksrow within 1 s. - Rust
hrrr-point-workerlogs show claim→complete within 30 s of insert. - HRRR profile appears on the QSO detail page within 1 min.
:hrrrOban queue is empty and can be removed from the Oban config.
Rollback Plans
Stream B: git revert restores MRMS. Supervision tree re-adds MrmsCache on next deploy. No data loss — MrmsCache was in-memory only.
Stream A: git revert restores the Elixir f00 chain step. The grid_tasks.kind column can stay (default 'forecast' keeps existing Rust behaviour). Bump hot pod memory back to 2Gi if Phase 3 shrink was already deployed.
Stream C: git revert restores HrrrFetchWorker. hrrr_point_tasks rows become orphan; a one-off Mix task migrates them back to Oban jobs.