Graham McIntire
f26e0dc226
Three independent improvements in one commit:
1. Parallel band scoring (pipeline.rs):
- rayon par_iter across 23 bands replaces the serial for loop.
- All band files now write via try_join_all over spawn_blocking
instead of serializing one-at-a-time on NFS. Chain per-step
drops from ~60s to ~15-25s on a 4-core box.
2. Prometheus metrics (new metrics.rs):
- axum server on METRICS_ADDR (default :9100) exposes /metrics and
/health. Scraped by existing Prometheus via annotation.
- Histograms: chain step duration (by outcome), decode duration.
- Gauge: tasks in flight (RAII guarded so panics don't leak).
- Counter: chain steps by outcome.
- worker.rs records step duration and wraps each run in an
InFlightGuard.
3. HA + ordering fixes:
- deployment-grid-rs.yaml: replicas 1→2, required podAntiAffinity
spreads them across hosts, nodeAffinity prefers talos5 for one.
PROP_GRID_RS_PARALLELISM 4→3 per pod (6 concurrent across
replicas; smaller secondary-node footprint).
- Readiness probe on /health so rollouts wait for the runtime to
be alive.
- claim_next ordering: run_time ASC → DESC so the newest hourly
run drains before any stragglers from a broken prior run. Within
a run, forecast_hour ASC keeps nearest-first.
- grid_task_enqueuer sets kind="forecast" explicitly and updates
conflict_target to the new 3-column unique index introduced by
migration 20260419222624.
61 lines
1.7 KiB
Elixir
61 lines
1.7 KiB
Elixir
defmodule Microwaveprop.Propagation.GridTaskEnqueuer do
|
|
@moduledoc """
|
|
Seeds `grid_tasks` rows for the Rust `prop-grid-rs` worker.
|
|
|
|
Called from `PropagationGridWorker.seed_chain/0` alongside the existing
|
|
Elixir f00..f18 Oban fan-out. Rust only claims rows with
|
|
`forecast_hour > 0`; Elixir still owns the f00 analysis-hour chain
|
|
because of native-duct + NEXRAD + commercial-link enrichment.
|
|
|
|
Inserts are idempotent via the `(run_time, forecast_hour)` unique
|
|
index — re-seeding the same cycle is a no-op.
|
|
"""
|
|
|
|
alias Microwaveprop.Repo
|
|
|
|
require Logger
|
|
|
|
@max_forecast_hour 18
|
|
|
|
@spec seed(DateTime.t()) :: {:ok, non_neg_integer()} | {:error, term()}
|
|
def seed(%DateTime{} = run_time) do
|
|
run_time = DateTime.truncate(run_time, :second)
|
|
now = DateTime.truncate(DateTime.utc_now(), :microsecond)
|
|
|
|
rows =
|
|
for fh <- 1..@max_forecast_hour do
|
|
%{
|
|
id: Ecto.UUID.bingenerate(),
|
|
run_time: run_time,
|
|
forecast_hour: fh,
|
|
valid_time: DateTime.add(run_time, fh * 3600, :second),
|
|
status: "queued",
|
|
attempt: 0,
|
|
kind: "forecast",
|
|
claimed_at: nil,
|
|
completed_at: nil,
|
|
error: nil,
|
|
inserted_at: now,
|
|
updated_at: now
|
|
}
|
|
end
|
|
|
|
{count, _} =
|
|
Repo.insert_all("grid_tasks", rows,
|
|
on_conflict: :nothing,
|
|
conflict_target: [:run_time, :forecast_hour, :kind]
|
|
)
|
|
|
|
if count > 0 do
|
|
Logger.info("GridTaskEnqueuer: seeded #{count} grid_tasks for run_time=#{run_time}")
|
|
else
|
|
Logger.info("GridTaskEnqueuer: run_time=#{run_time} already seeded")
|
|
end
|
|
|
|
{:ok, count}
|
|
rescue
|
|
e ->
|
|
Logger.error("GridTaskEnqueuer: seed failed: #{inspect(e)}")
|
|
{:error, e}
|
|
end
|
|
end
|