Graham McIntire
5ec66df135
The scoring+upsert phase was ~4m40s per forecast hour and dominated wall time. Three stacked optimizations attack it from different angles. replace_scores/2 is a new hot-path writer that does DELETE WHERE valid_time = $1 followed by a plain insert_all (no ON CONFLICT resolution). The chain worker rewrites the full (valid_time, all bands) slice every forecast hour, so conflict detection was pure waste. AsosAdjustmentWorker still uses upsert_scores because it only rewrites the subset of cells near a station. factors is now nullable. Forecast hours f01-f18 pass factors: nil so the JSONB encode + toast write is skipped entirely — roughly halves the data volume per run. point_detail/4 coalesces nil to an empty map so the JS popup renders without a TypeError, and scorer_diff only pulls the most recent valid_time that still has factors (the f00 row). propagation_scores is now UNLOGGED, so inserts bypass WAL entirely. Durability tradeoff: an unclean shutdown truncates the table, but PropagationGridWorker rebuilds it from HRRR every 3h so a lost table is re-populated within one cron cycle. Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a speculative plan for a flat-file / DuckDB rewrite with explicit trigger conditions for when to pick it up (partitioning deferred too; revisit only if these three don't solve it).
32 lines
822 B
Elixir
32 lines
822 B
Elixir
defmodule Microwaveprop.Propagation.GridScore do
|
|
@moduledoc false
|
|
use Ecto.Schema
|
|
|
|
import Ecto.Changeset
|
|
|
|
@primary_key {:id, :binary_id, autogenerate: true}
|
|
|
|
schema "propagation_scores" do
|
|
field :lat, :float
|
|
field :lon, :float
|
|
field :valid_time, :utc_datetime
|
|
field :band_mhz, :integer
|
|
field :score, :integer
|
|
field :factors, :map
|
|
|
|
timestamps(type: :utc_datetime)
|
|
end
|
|
|
|
@type t :: %__MODULE__{}
|
|
|
|
@fields ~w(lat lon valid_time band_mhz score factors)a
|
|
@required_fields ~w(lat lon valid_time band_mhz score)a
|
|
|
|
@spec changeset(t() | Ecto.Changeset.t(), map()) :: Ecto.Changeset.t()
|
|
def changeset(grid_score, attrs) do
|
|
grid_score
|
|
|> cast(attrs, @fields)
|
|
|> validate_required(@required_fields)
|
|
|> unique_constraint([:lat, :lon, :valid_time, :band_mhz])
|
|
end
|
|
end
|