Graham McIntire
7416583c27
Phase 3 Stream C Elixir-side: HrrrFetchWorker is deleted; per-QSO HRRR
enrichment now writes to the new hrrr_fetch_tasks table for the Rust
hrrr-point-worker to drain.
Table shape:
- one row per valid_time (primary key) with a JSONB array of
{lat, lon} points
- UPSERT on conflict: array-union of points, status flips back to
queued if previously done/failed so a backfill re-scan naturally
refills the queue for Rust
Elixir changes:
- new migration 20260419231502_create_hrrr_fetch_tasks
- new Microwaveprop.Weather.HrrrPointEnqueuer with enqueue/1 and
enqueue_for_contacts/1. Pre-2014 contacts (NARR's territory)
are skipped here so hrrr_status can pin them to :unavailable.
- ContactWeatherEnqueueWorker: build_hrrr_jobs/1 removed; single-
contact path and batch perform/1 both route through
HrrrPointEnqueuer.enqueue_for_contacts/1. A placeholder jobs-list
is kept just to feed mark_hrrr_status!.
- contact_live/show.ex retry button enqueues via the same path.
- :hrrr queue removed from dev/config/runtime.exs
- HrrrFetchWorker module + test deleted
BackfillEnqueueWorker scans continue to flow through
ContactWeatherEnqueueWorker.enqueue_for_contact (unchanged), so the
30-min reconcile refills hrrr_fetch_tasks automatically.
4 new tests cover the routing, pre-2014 skip, UPSERT-union, and
status-reset-on-reschedule behaviour. Rust-side hrrr-point-worker
binary + k8s deployment land in the next commits.
43 lines
1.7 KiB
Elixir
43 lines
1.7 KiB
Elixir
defmodule Microwaveprop.Repo.Migrations.CreateHrrrFetchTasks do
|
|
use Ecto.Migration
|
|
|
|
# Hand-off queue between the Elixir enrichment enqueuer and the Rust
|
|
# hrrr-point-worker (Phase 3 Stream C). One row per valid_time with a
|
|
# JSONB array of {lat, lon} points the workers needs to extract from
|
|
# that hour's HRRR cycle. Rust claims by SKIP LOCKED, fetches the
|
|
# GRIB2 once, extracts every point in the batch, bulk-upserts to
|
|
# hrrr_profiles, and marks the row done.
|
|
def change do
|
|
create table(:hrrr_fetch_tasks, primary_key: false) do
|
|
add :id, :binary_id, primary_key: true
|
|
add :valid_time, :utc_datetime, null: false
|
|
|
|
# [{"lat": f, "lon": f}, …] — additional points for the same
|
|
# valid_time get merged via jsonb array-union UPSERT so a
|
|
# backfill tick doesn't schedule a new row per contact.
|
|
add :points, :map, null: false, default: fragment("'[]'::jsonb")
|
|
|
|
add :status, :string, null: false, default: "queued"
|
|
add :attempt, :integer, null: false, default: 0
|
|
add :claimed_at, :utc_datetime_usec
|
|
add :completed_at, :utc_datetime_usec
|
|
add :error, :text
|
|
|
|
timestamps(type: :utc_datetime)
|
|
end
|
|
|
|
# One task per valid_time. Merges all points for the hour into a
|
|
# single fetch so a full backfill scan costs one GRIB2 download
|
|
# per hour, not one per contact.
|
|
create unique_index(:hrrr_fetch_tasks, [:valid_time])
|
|
|
|
# Hot-path claim query scans the queued subset.
|
|
create index(:hrrr_fetch_tasks, [:status],
|
|
where: "status = 'queued'",
|
|
name: :hrrr_fetch_tasks_queued_idx
|
|
)
|
|
|
|
# Prune index for retention.
|
|
create index(:hrrr_fetch_tasks, [:completed_at])
|
|
end
|
|
end
|