Graham McIntire
b80878056d
Extracts the memory-hostile HRRR fetch → decode → score pipeline for
forecast hours 1–18 into a separate Rust service (`prop-grid-rs`).
Elixir retains f00 with its native-duct + NEXRAD + commercial-link
enrichment; Rust handles the other 18 steps per hourly chain.
Hand-off via a new `grid_tasks` table (`FOR UPDATE SKIP LOCKED` claim
from Rust, Postgrex `NOTIFY propagation_ready` back to Elixir). Rust
writes the same on-disk score-grid format Elixir already uses, to the
same `/data/scores` NFS tree. Phase 1 ships in shadow mode with
PROP_SCORES_DIR=/data/scores_shadow.
Rust crate layout at `rust/prop_grid_rs/` (1:1 module parity with the
Elixir source it ports):
- grid, region, band_config, scores_file, sounding_params
- scorer: all 10 factors + composite. Matches the Elixir scorer
byte-for-byte across 115 golden-fixture samples (5 scenarios
× 23 bands).
- decoder: wgrib2 subprocess + Fortran-record lola-binary parser
- fetcher: HRRR URL derivation + idx cache (1h TTL) + 8-way
parallel byte-range downloads with 429/5xx retry
- pipeline: end-to-end chain step, f00 rejected at the boundary
- db: sqlx grid_tasks claim/complete + propagation_ready NOTIFY
- bin/worker: tokio main loop, JSON logs, SIGTERM-safe
91 Rust tests + clippy -D warnings clean. 2,158 Elixir tests green.
Elixir additions:
- `GridTaskEnqueuer` seeds fh=1..18 rows from
`PropagationGridWorker.seed_chain/0`
- `NotifyListener` LISTEN → warm `ScoreCache` → PubSub fan-out
- `ShadowComparator` diffs prod vs shadow `.ntms` bodies
- `mix rust.golden` task writes the Rust-side golden fixture
k8s: new `deployment-grid-rs.yaml` pinned to talos5 (32 GB NUC) via
`prop-grid-rs=primary` nodeSelector + `workload=grid-rs:NoSchedule`
toleration, 512 Mi limit, sharing the existing NFS `/data` mount.
The plan document is at `plans/vivid-hatching-quail.md` (local to my
workstation); phases 2 (cutover) and 3 (talos5 concurrency tuning)
follow after 72h of shadow-mode parity.
48 lines
2.2 KiB
Elixir
48 lines
2.2 KiB
Elixir
defmodule Microwaveprop.Repo.Migrations.CreateGridTasks do
|
|
use Ecto.Migration
|
|
|
|
def change do
|
|
# Hand-off queue between Elixir's PropagationGridWorker (seed) and the
|
|
# external Rust `prop-grid-rs` worker. Each row is one forecast hour's
|
|
# worth of fetch → decode → score → write-ntms work. Rust claims with
|
|
# `SELECT ... FOR UPDATE SKIP LOCKED` on (status='queued'), marks
|
|
# 'running' while working, and transitions to 'done' / 'failed' on
|
|
# completion. On 'done' it NOTIFYs `propagation_ready` so Elixir pods
|
|
# can warm ScoreCache without polling.
|
|
create table(:grid_tasks, primary_key: false) do
|
|
add :id, :binary_id, primary_key: true
|
|
|
|
# HRRR cycle + step. (run_time, forecast_hour) is the natural key; an
|
|
# hourly cron re-enqueueing the same step is a no-op via the unique
|
|
# index rather than a conflict that has to be handled.
|
|
add :run_time, :utc_datetime, null: false
|
|
add :forecast_hour, :integer, null: false
|
|
add :valid_time, :utc_datetime, null: false
|
|
|
|
# 'queued' → 'running' → 'done' | 'failed'. Not an enum so we can add
|
|
# more states later without a migration.
|
|
add :status, :string, null: false, default: "queued"
|
|
|
|
# How many times a worker has claimed this row. Incremented on each
|
|
# claim so runaway retries are observable without scanning logs.
|
|
add :attempt, :integer, null: false, default: 0
|
|
|
|
add :claimed_at, :utc_datetime_usec
|
|
add :completed_at, :utc_datetime_usec
|
|
add :error, :text
|
|
|
|
timestamps(type: :utc_datetime)
|
|
end
|
|
|
|
# Uniqueness on the natural key lets the seeder run `INSERT ... ON CONFLICT
|
|
# DO NOTHING` and stay idempotent across retries of the seed job.
|
|
create unique_index(:grid_tasks, [:run_time, :forecast_hour])
|
|
|
|
# Rust's claim query filters on status first. A partial index on just the
|
|
# claimable subset keeps the common lookup cheap as finished rows
|
|
# accumulate — the table stays small (19 rows per hour, pruned on a cron)
|
|
# but the principle is cheap either way.
|
|
create index(:grid_tasks, [:status], where: "status = 'queued'", name: :grid_tasks_queued_idx)
|
|
create index(:grid_tasks, [:completed_at])
|
|
end
|
|
end
|