defmodule Microwaveprop.Repo.Migrations.CreateGridTasks do use Ecto.Migration def change do # Hand-off queue between Elixir's PropagationGridWorker (seed) and the # external Rust `prop-grid-rs` worker. Each row is one forecast hour's # worth of fetch → decode → score → write-ntms work. Rust claims with # `SELECT ... FOR UPDATE SKIP LOCKED` on (status='queued'), marks # 'running' while working, and transitions to 'done' / 'failed' on # completion. On 'done' it NOTIFYs `propagation_ready` so Elixir pods # can warm ScoreCache without polling. create table(:grid_tasks, primary_key: false) do add :id, :binary_id, primary_key: true # HRRR cycle + step. (run_time, forecast_hour) is the natural key; an # hourly cron re-enqueueing the same step is a no-op via the unique # index rather than a conflict that has to be handled. add :run_time, :utc_datetime, null: false add :forecast_hour, :integer, null: false add :valid_time, :utc_datetime, null: false # 'queued' → 'running' → 'done' | 'failed'. Not an enum so we can add # more states later without a migration. add :status, :string, null: false, default: "queued" # How many times a worker has claimed this row. Incremented on each # claim so runaway retries are observable without scanning logs. add :attempt, :integer, null: false, default: 0 add :claimed_at, :utc_datetime_usec add :completed_at, :utc_datetime_usec add :error, :text timestamps(type: :utc_datetime) end # Uniqueness on the natural key lets the seeder run `INSERT ... ON CONFLICT # DO NOTHING` and stay idempotent across retries of the seed job. create unique_index(:grid_tasks, [:run_time, :forecast_hour]) # Rust's claim query filters on status first. A partial index on just the # claimable subset keeps the common lookup cheap as finished rows # accumulate — the table stays small (19 rows per hour, pruned on a cron) # but the principle is cheap either way. create index(:grid_tasks, [:status], where: "status = 'queued'", name: :grid_tasks_queued_idx) create index(:grid_tasks, [:completed_at]) end end