feat(pskr): hourly calibration sampler joining spots × HRRR × Kp
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
This commit is contained in:
parent
7702b9e161
commit
7a2b1f292c
7 changed files with 690 additions and 1 deletions
|
|
@ -309,7 +309,14 @@ if config_env() == :prod do
|
||||||
# cheap enough to run daily so the table self-heals if a
|
# cheap enough to run daily so the table self-heals if a
|
||||||
# deploy clears it or new profiles need to fold into the means.
|
# deploy clears it or new profiles need to fold into the means.
|
||||||
# 02:30 UTC sits inside the quiet pre-NARR/HRRR ingestion gap.
|
# 02:30 UTC sits inside the quiet pre-NARR/HRRR ingestion gap.
|
||||||
{"30 2 * * *", Microwaveprop.Workers.AdminTaskWorker, args: %{"task" => "climatology"}}
|
{"30 2 * * *", Microwaveprop.Workers.AdminTaskWorker, args: %{"task" => "climatology"}},
|
||||||
|
# Build the PSKR calibration corpus one hour at a time. HH:25
|
||||||
|
# is past the HRRR analysis publish window (~HH:50 of the
|
||||||
|
# previous hour, surfaces by HH:05) and past PSKR's 60s
|
||||||
|
# aggregator flush, so the previous-hour sample has the
|
||||||
|
# atmospheric and signal data it needs. Idempotent UPSERT, so
|
||||||
|
# missed fires are safe to backfill manually.
|
||||||
|
{"25 * * * *", Microwaveprop.Workers.PskrCalibrationWorker}
|
||||||
]}
|
]}
|
||||||
|
|
||||||
base_plugins = [
|
base_plugins = [
|
||||||
|
|
|
||||||
80
lib/microwaveprop/pskr/calibration_sample.ex
Normal file
80
lib/microwaveprop/pskr/calibration_sample.ex
Normal file
|
|
@ -0,0 +1,80 @@
|
||||||
|
defmodule Microwaveprop.Pskr.CalibrationSample do
|
||||||
|
@moduledoc """
|
||||||
|
One sample for the PSKR-driven recalibration corpus: a (hour, band,
|
||||||
|
midpoint cell) bucket joining aggregate PSKR spot density with the
|
||||||
|
HRRR atmospheric state and SWPC geomagnetic state at that hour.
|
||||||
|
|
||||||
|
## Why we don't store predicted scores
|
||||||
|
|
||||||
|
The corpus exists so the recalibrator can learn weights from
|
||||||
|
(atmospheric features → observed spot density). Predicted scores
|
||||||
|
are derived quantities that change every time the algorithm
|
||||||
|
changes; storing them locks the corpus to a single algorithm
|
||||||
|
version. Features are immutable — they describe what the
|
||||||
|
atmosphere actually was — so the same sample row stays useful
|
||||||
|
across every weight refit.
|
||||||
|
|
||||||
|
## Cell granularity
|
||||||
|
|
||||||
|
`midpoint_lat`/`midpoint_lon` are snapped to the propagation grid
|
||||||
|
(currently 0.125°) by `Pskr.CalibrationSampler` before insert.
|
||||||
|
Two PSKR paths whose midpoints land in the same grid cell collapse
|
||||||
|
into a single row — the same logic that already drives
|
||||||
|
`pskr_spots_hourly` aggregation, applied at the spatial dimension.
|
||||||
|
|
||||||
|
## Forward-only
|
||||||
|
|
||||||
|
PSK Reporter has no historical archive — the firehose is real-time
|
||||||
|
only. The corpus grows from when the feed first started and never
|
||||||
|
backfills further. Recalibration weight refits should only be
|
||||||
|
attempted once enough days have accumulated to cover diurnal and
|
||||||
|
synoptic variability (heuristic: ≥ 30 d, ≥ 10k samples).
|
||||||
|
"""
|
||||||
|
use Ecto.Schema
|
||||||
|
|
||||||
|
import Ecto.Changeset
|
||||||
|
|
||||||
|
@primary_key {:id, :binary_id, autogenerate: true}
|
||||||
|
@foreign_key_type :binary_id
|
||||||
|
|
||||||
|
schema "pskr_calibration_samples" do
|
||||||
|
field :hour_utc, :utc_datetime
|
||||||
|
field :band, :string
|
||||||
|
field :midpoint_lat, :float
|
||||||
|
field :midpoint_lon, :float
|
||||||
|
|
||||||
|
field :spot_count, :integer, default: 0
|
||||||
|
field :distinct_paths, :integer, default: 0
|
||||||
|
field :median_distance_km, :float
|
||||||
|
field :modes, {:array, :string}, default: []
|
||||||
|
|
||||||
|
field :surface_temp_c, :float
|
||||||
|
field :surface_dewpoint_c, :float
|
||||||
|
field :pwat_mm, :float
|
||||||
|
field :surface_pressure_mb, :float
|
||||||
|
field :min_refractivity_gradient, :float
|
||||||
|
field :hpbl_m, :float
|
||||||
|
field :hrrr_ducting_detected, :boolean
|
||||||
|
|
||||||
|
field :kp_index, :integer
|
||||||
|
|
||||||
|
timestamps(type: :utc_datetime)
|
||||||
|
end
|
||||||
|
|
||||||
|
@type t :: %__MODULE__{}
|
||||||
|
|
||||||
|
@cast_fields ~w(hour_utc band midpoint_lat midpoint_lon spot_count distinct_paths
|
||||||
|
median_distance_km modes surface_temp_c surface_dewpoint_c pwat_mm
|
||||||
|
surface_pressure_mb min_refractivity_gradient hpbl_m hrrr_ducting_detected
|
||||||
|
kp_index)a
|
||||||
|
|
||||||
|
@required_fields ~w(hour_utc band midpoint_lat midpoint_lon spot_count)a
|
||||||
|
|
||||||
|
@spec changeset(t(), map()) :: Ecto.Changeset.t()
|
||||||
|
def changeset(record, attrs) do
|
||||||
|
record
|
||||||
|
|> cast(attrs, @cast_fields)
|
||||||
|
|> validate_required(@required_fields)
|
||||||
|
|> unique_constraint([:hour_utc, :band, :midpoint_lat, :midpoint_lon])
|
||||||
|
end
|
||||||
|
end
|
||||||
236
lib/microwaveprop/pskr/calibration_sampler.ex
Normal file
236
lib/microwaveprop/pskr/calibration_sampler.ex
Normal file
|
|
@ -0,0 +1,236 @@
|
||||||
|
defmodule Microwaveprop.Pskr.CalibrationSampler do
|
||||||
|
@moduledoc """
|
||||||
|
Builds `Pskr.CalibrationSample` rows by joining PSKR spot density
|
||||||
|
with HRRR atmospheric state and SWPC geomagnetic state at a given
|
||||||
|
hour.
|
||||||
|
|
||||||
|
Workflow:
|
||||||
|
|
||||||
|
1. Pull every `pskr_spots_hourly` row for the target hour.
|
||||||
|
2. Snap each row's midpoint to the propagation grid (0.125°) and
|
||||||
|
bucket by `(band, snapped_lat, snapped_lon)`.
|
||||||
|
3. For each bucket, aggregate the spot stats (sum count, count
|
||||||
|
distinct paths, dedup modes, median distance).
|
||||||
|
4. For each bucket, look up the nearest `hrrr_profiles` row to the
|
||||||
|
midpoint at the target hour and copy the surface/refractivity
|
||||||
|
fields onto the sample. HRRR is pulled in one window query and
|
||||||
|
joined in-memory to keep the per-bucket cost a hashtable lookup.
|
||||||
|
5. Look up SWPC Kp once per run.
|
||||||
|
6. Bulk-upsert the result with `(hour_utc, band, midpoint_lat,
|
||||||
|
midpoint_lon)` as the conflict target.
|
||||||
|
|
||||||
|
The sampler is intentionally idempotent: re-running for the same
|
||||||
|
hour is a no-op except for the `updated_at` bump, so the worker
|
||||||
|
can safely overlap with manual reruns and the cron retry on
|
||||||
|
failure.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import Ecto.Query
|
||||||
|
|
||||||
|
alias Microwaveprop.Pskr.CalibrationSample
|
||||||
|
alias Microwaveprop.Pskr.SpotHourly
|
||||||
|
alias Microwaveprop.Repo
|
||||||
|
alias Microwaveprop.SpaceWeather
|
||||||
|
alias Microwaveprop.Weather.HrrrProfile
|
||||||
|
|
||||||
|
require Logger
|
||||||
|
|
||||||
|
@grid_step_deg 0.125
|
||||||
|
# HRRR domain is ~3 km native, so a ±0.07° (~7 km) lat/lon window
|
||||||
|
# plus ±60 min reliably finds at least one profile for any CONUS
|
||||||
|
# midpoint at most hours.
|
||||||
|
@hrrr_lookahead_deg 0.07
|
||||||
|
@hrrr_lookahead_seconds 3600
|
||||||
|
|
||||||
|
@doc """
|
||||||
|
Build calibration samples for one hour. Returns the upsert count.
|
||||||
|
Returns `0` when no PSKR spots were captured for that hour — for
|
||||||
|
example, during PSKR client downtime.
|
||||||
|
"""
|
||||||
|
@spec build_for_hour(DateTime.t()) :: non_neg_integer()
|
||||||
|
def build_for_hour(%DateTime{} = hour_utc) do
|
||||||
|
hour_utc = align_to_hour(hour_utc)
|
||||||
|
spots = fetch_spots(hour_utc)
|
||||||
|
|
||||||
|
if spots == [] do
|
||||||
|
Logger.info("Pskr.CalibrationSampler: no spots for #{hour_utc} — skipping")
|
||||||
|
0
|
||||||
|
else
|
||||||
|
hrrr_index = build_hrrr_index(hour_utc)
|
||||||
|
kp = fetch_kp(hour_utc)
|
||||||
|
|
||||||
|
samples =
|
||||||
|
spots
|
||||||
|
|> bucket_by_cell()
|
||||||
|
|> Enum.map(&build_sample(&1, hour_utc, hrrr_index, kp))
|
||||||
|
|
||||||
|
upsert(samples)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# ── Spot fetch + cell bucketing ─────────────────────────────────
|
||||||
|
|
||||||
|
defp fetch_spots(hour_utc) do
|
||||||
|
Repo.all(
|
||||||
|
from(s in SpotHourly,
|
||||||
|
where: s.hour_utc == ^hour_utc and not is_nil(s.midpoint_lat) and not is_nil(s.midpoint_lon),
|
||||||
|
select: %{
|
||||||
|
band: s.band,
|
||||||
|
midpoint_lat: s.midpoint_lat,
|
||||||
|
midpoint_lon: s.midpoint_lon,
|
||||||
|
spot_count: s.spot_count,
|
||||||
|
distance_km: s.distance_km,
|
||||||
|
modes: s.modes
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
end
|
||||||
|
|
||||||
|
defp bucket_by_cell(spots) do
|
||||||
|
Enum.group_by(spots, fn s -> {s.band, snap(s.midpoint_lat), snap(s.midpoint_lon)} end)
|
||||||
|
end
|
||||||
|
|
||||||
|
defp snap(deg), do: Float.round(deg / @grid_step_deg) * @grid_step_deg
|
||||||
|
|
||||||
|
# ── HRRR feature lookup ─────────────────────────────────────────
|
||||||
|
|
||||||
|
defp build_hrrr_index(hour_utc) do
|
||||||
|
time_start = DateTime.add(hour_utc, -@hrrr_lookahead_seconds, :second)
|
||||||
|
time_end = DateTime.add(hour_utc, @hrrr_lookahead_seconds, :second)
|
||||||
|
|
||||||
|
Repo.all(
|
||||||
|
from(h in HrrrProfile,
|
||||||
|
where: h.valid_time >= ^time_start and h.valid_time <= ^time_end,
|
||||||
|
select: %{
|
||||||
|
lat: h.lat,
|
||||||
|
lon: h.lon,
|
||||||
|
surface_temp_c: h.surface_temp_c,
|
||||||
|
surface_dewpoint_c: h.surface_dewpoint_c,
|
||||||
|
pwat_mm: h.pwat_mm,
|
||||||
|
surface_pressure_mb: h.surface_pressure_mb,
|
||||||
|
min_refractivity_gradient: h.min_refractivity_gradient,
|
||||||
|
hpbl_m: h.hpbl_m,
|
||||||
|
ducting_detected: h.ducting_detected
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
end
|
||||||
|
|
||||||
|
# Pick the HRRR row whose lat/lon is closest to (lat, lon). Linear
|
||||||
|
# scan over the in-memory window — at the typical ~5k profile
|
||||||
|
# window size and ~1k cells per hour, this is faster than 1k DB
|
||||||
|
# round trips. Returns `nil` if no profile is in the window.
|
||||||
|
defp nearest_hrrr(_lat, _lon, []), do: nil
|
||||||
|
|
||||||
|
defp nearest_hrrr(lat, lon, profiles) do
|
||||||
|
candidates =
|
||||||
|
Enum.filter(profiles, fn p ->
|
||||||
|
abs(p.lat - lat) <= @hrrr_lookahead_deg and abs(p.lon - lon) <= @hrrr_lookahead_deg
|
||||||
|
end)
|
||||||
|
|
||||||
|
case candidates do
|
||||||
|
[] -> nil
|
||||||
|
list -> Enum.min_by(list, fn p -> abs(p.lat - lat) + abs(p.lon - lon) end)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# ── Kp lookup ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
# Single Kp value applies cluster-wide for the hour. We prefer the
|
||||||
|
# integer `kp_index` (the official 3-hour Kp) and fall back to the
|
||||||
|
# truncated `estimated_kp` when SWPC hasn't published the official
|
||||||
|
# value yet — same precedence as `Propagation.current_kp_index/0`.
|
||||||
|
defp fetch_kp(_hour_utc) do
|
||||||
|
case SpaceWeather.latest_kp() do
|
||||||
|
%{kp_index: kp} when is_integer(kp) -> kp
|
||||||
|
%{estimated_kp: kp} when is_number(kp) -> trunc(kp)
|
||||||
|
_ -> nil
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# ── Per-bucket sample assembly ──────────────────────────────────
|
||||||
|
|
||||||
|
defp build_sample({{band, lat, lon}, cell_spots}, hour_utc, hrrr_index, kp) do
|
||||||
|
hrrr = nearest_hrrr(lat, lon, hrrr_index)
|
||||||
|
now = DateTime.truncate(DateTime.utc_now(), :second)
|
||||||
|
|
||||||
|
%{
|
||||||
|
id: Ecto.UUID.generate(),
|
||||||
|
hour_utc: hour_utc,
|
||||||
|
band: band,
|
||||||
|
midpoint_lat: lat,
|
||||||
|
midpoint_lon: lon,
|
||||||
|
spot_count: Enum.sum(Enum.map(cell_spots, & &1.spot_count)),
|
||||||
|
distinct_paths: length(cell_spots),
|
||||||
|
median_distance_km: median_distance(cell_spots),
|
||||||
|
modes: dedup_modes(cell_spots),
|
||||||
|
surface_temp_c: hrrr && hrrr.surface_temp_c,
|
||||||
|
surface_dewpoint_c: hrrr && hrrr.surface_dewpoint_c,
|
||||||
|
pwat_mm: hrrr && hrrr.pwat_mm,
|
||||||
|
surface_pressure_mb: hrrr && hrrr.surface_pressure_mb,
|
||||||
|
min_refractivity_gradient: hrrr && hrrr.min_refractivity_gradient,
|
||||||
|
hpbl_m: hrrr && hrrr.hpbl_m,
|
||||||
|
hrrr_ducting_detected: hrrr && hrrr.ducting_detected,
|
||||||
|
kp_index: kp,
|
||||||
|
inserted_at: now,
|
||||||
|
updated_at: now
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
defp median_distance(cell_spots) do
|
||||||
|
cell_spots
|
||||||
|
|> Enum.map(& &1.distance_km)
|
||||||
|
|> Enum.reject(&is_nil/1)
|
||||||
|
|> case do
|
||||||
|
[] ->
|
||||||
|
nil
|
||||||
|
|
||||||
|
list ->
|
||||||
|
sorted = Enum.sort(list)
|
||||||
|
n = length(sorted)
|
||||||
|
mid = div(n, 2)
|
||||||
|
if rem(n, 2) == 0, do: (Enum.at(sorted, mid - 1) + Enum.at(sorted, mid)) / 2, else: Enum.at(sorted, mid)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
defp dedup_modes(cell_spots) do
|
||||||
|
cell_spots |> Enum.flat_map(& &1.modes) |> Enum.uniq()
|
||||||
|
end
|
||||||
|
|
||||||
|
# ── Upsert ──────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
defp upsert([]), do: 0
|
||||||
|
|
||||||
|
defp upsert(samples) do
|
||||||
|
{count, _} =
|
||||||
|
Repo.insert_all(CalibrationSample, samples,
|
||||||
|
on_conflict:
|
||||||
|
from(c in CalibrationSample,
|
||||||
|
update: [
|
||||||
|
set: [
|
||||||
|
spot_count: fragment("EXCLUDED.spot_count"),
|
||||||
|
distinct_paths: fragment("EXCLUDED.distinct_paths"),
|
||||||
|
median_distance_km: fragment("EXCLUDED.median_distance_km"),
|
||||||
|
modes: fragment("EXCLUDED.modes"),
|
||||||
|
surface_temp_c: fragment("EXCLUDED.surface_temp_c"),
|
||||||
|
surface_dewpoint_c: fragment("EXCLUDED.surface_dewpoint_c"),
|
||||||
|
pwat_mm: fragment("EXCLUDED.pwat_mm"),
|
||||||
|
surface_pressure_mb: fragment("EXCLUDED.surface_pressure_mb"),
|
||||||
|
min_refractivity_gradient: fragment("EXCLUDED.min_refractivity_gradient"),
|
||||||
|
hpbl_m: fragment("EXCLUDED.hpbl_m"),
|
||||||
|
hrrr_ducting_detected: fragment("EXCLUDED.hrrr_ducting_detected"),
|
||||||
|
kp_index: fragment("EXCLUDED.kp_index"),
|
||||||
|
updated_at: fragment("EXCLUDED.updated_at")
|
||||||
|
]
|
||||||
|
]
|
||||||
|
),
|
||||||
|
conflict_target: [:hour_utc, :band, :midpoint_lat, :midpoint_lon]
|
||||||
|
)
|
||||||
|
|
||||||
|
count
|
||||||
|
end
|
||||||
|
|
||||||
|
defp align_to_hour(%DateTime{} = dt) do
|
||||||
|
%{dt | minute: 0, second: 0, microsecond: {0, 0}}
|
||||||
|
end
|
||||||
|
end
|
||||||
56
lib/microwaveprop/workers/pskr_calibration_worker.ex
Normal file
56
lib/microwaveprop/workers/pskr_calibration_worker.ex
Normal file
|
|
@ -0,0 +1,56 @@
|
||||||
|
defmodule Microwaveprop.Workers.PskrCalibrationWorker do
|
||||||
|
@moduledoc """
|
||||||
|
Hourly cron that builds `Pskr.CalibrationSample` rows by joining
|
||||||
|
PSKR spot density with HRRR atmospheric state and SWPC space
|
||||||
|
weather at a given hour.
|
||||||
|
|
||||||
|
Default behaviour fires for the previous full hour — by HH:25 UTC
|
||||||
|
the HRRR cycle has analysis available and PSKR's hourly aggregator
|
||||||
|
has flushed its accumulator. Pass `"hour_utc"` in the args map to
|
||||||
|
rerun a specific hour:
|
||||||
|
|
||||||
|
Microwaveprop.Workers.PskrCalibrationWorker.new(%{"hour_utc" => "2026-05-04T18:00:00Z"})
|
||||||
|
|
||||||
|
The sampler is idempotent (UPSERT on `(hour_utc, band, midpoint_lat,
|
||||||
|
midpoint_lon)`) so reruns are safe.
|
||||||
|
|
||||||
|
Lives on the `:backfill_enqueue` queue rather than spawning a new
|
||||||
|
one — the workload is small (~1 query window, in-memory join,
|
||||||
|
bulk upsert) and cluster-wide single-slot is plenty for one fire
|
||||||
|
per hour.
|
||||||
|
"""
|
||||||
|
use Oban.Worker,
|
||||||
|
queue: :backfill_enqueue,
|
||||||
|
max_attempts: 3,
|
||||||
|
unique: [
|
||||||
|
period: 3600,
|
||||||
|
states: [:available, :scheduled, :executing, :retryable]
|
||||||
|
]
|
||||||
|
|
||||||
|
alias Microwaveprop.Pskr.CalibrationSampler
|
||||||
|
|
||||||
|
require Logger
|
||||||
|
|
||||||
|
@impl Oban.Worker
|
||||||
|
def perform(%Oban.Job{args: args}) do
|
||||||
|
hour = parse_hour(args)
|
||||||
|
Logger.info("PskrCalibrationWorker: building samples for #{hour}")
|
||||||
|
count = CalibrationSampler.build_for_hour(hour)
|
||||||
|
Logger.info("PskrCalibrationWorker: upserted #{count} samples for #{hour}")
|
||||||
|
:ok
|
||||||
|
end
|
||||||
|
|
||||||
|
defp parse_hour(%{"hour_utc" => iso}) when is_binary(iso) do
|
||||||
|
{:ok, dt, _} = DateTime.from_iso8601(iso)
|
||||||
|
dt
|
||||||
|
end
|
||||||
|
|
||||||
|
defp parse_hour(_) do
|
||||||
|
# Default: the previous fully-completed hour.
|
||||||
|
DateTime.utc_now()
|
||||||
|
|> DateTime.add(-3600, :second)
|
||||||
|
|> Map.put(:minute, 0)
|
||||||
|
|> Map.put(:second, 0)
|
||||||
|
|> Map.put(:microsecond, {0, 0})
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
defmodule Microwaveprop.Repo.Migrations.CreatePskrCalibrationSamples do
|
||||||
|
use Ecto.Migration
|
||||||
|
|
||||||
|
# Calibration corpus joining PSKR spot density (truth signal) with
|
||||||
|
# HRRR atmospheric state and SWPC space weather (predictors). One
|
||||||
|
# row per (hour, band, midpoint cell). The recalibrator reads from
|
||||||
|
# this table and computes predicted scores on demand from the
|
||||||
|
# stored features — never store the prediction itself, since it
|
||||||
|
# depends on the algorithm version under evaluation.
|
||||||
|
|
||||||
|
def change do
|
||||||
|
create table(:pskr_calibration_samples, primary_key: false) do
|
||||||
|
add :id, :binary_id, primary_key: true
|
||||||
|
add :hour_utc, :utc_datetime, null: false
|
||||||
|
add :band, :string, null: false
|
||||||
|
add :midpoint_lat, :float, null: false
|
||||||
|
add :midpoint_lon, :float, null: false
|
||||||
|
|
||||||
|
# Aggregate PSKR signal at this cell.
|
||||||
|
add :spot_count, :integer, null: false, default: 0
|
||||||
|
add :distinct_paths, :integer, null: false, default: 0
|
||||||
|
add :median_distance_km, :float
|
||||||
|
add :modes, {:array, :string}, null: false, default: []
|
||||||
|
|
||||||
|
# HRRR atmospheric snapshot at midpoint at the hour boundary.
|
||||||
|
# Nullable because the join may miss for pre-2014 cells or
|
||||||
|
# cells outside the HRRR domain.
|
||||||
|
add :surface_temp_c, :float
|
||||||
|
add :surface_dewpoint_c, :float
|
||||||
|
add :pwat_mm, :float
|
||||||
|
add :surface_pressure_mb, :float
|
||||||
|
add :min_refractivity_gradient, :float
|
||||||
|
add :hpbl_m, :float
|
||||||
|
add :hrrr_ducting_detected, :boolean
|
||||||
|
|
||||||
|
# SWPC geomagnetic state at the hour boundary. Same Kp value
|
||||||
|
# applies to every cell in the same hour, but stored per-row so
|
||||||
|
# the recalibrator doesn't have to re-join SpaceWeather for
|
||||||
|
# every batch of samples.
|
||||||
|
add :kp_index, :integer
|
||||||
|
|
||||||
|
timestamps(type: :utc_datetime)
|
||||||
|
end
|
||||||
|
|
||||||
|
# Same uniqueness shape as `pskr_spots_hourly` so the upsert
|
||||||
|
# pattern translates 1:1. Cell granularity is set by the sampler
|
||||||
|
# (currently 0.125° to match the propagation grid).
|
||||||
|
create unique_index(:pskr_calibration_samples, [
|
||||||
|
:hour_utc,
|
||||||
|
:band,
|
||||||
|
:midpoint_lat,
|
||||||
|
:midpoint_lon
|
||||||
|
])
|
||||||
|
|
||||||
|
create index(:pskr_calibration_samples, [:hour_utc])
|
||||||
|
|
||||||
|
# Make the spot-side spatial filter the sampler runs cheap.
|
||||||
|
# `pskr_spots_hourly` already has unique-index coverage on
|
||||||
|
# (hour_utc, band, sender_grid, receiver_grid) but no index that
|
||||||
|
# supports a midpoint range scan, which is the hot path for both
|
||||||
|
# the sampler join and any future map-side density queries.
|
||||||
|
create index(:pskr_spots_hourly, [:midpoint_lat, :midpoint_lon])
|
||||||
|
end
|
||||||
|
end
|
||||||
184
test/microwaveprop/pskr/calibration_sampler_test.exs
Normal file
184
test/microwaveprop/pskr/calibration_sampler_test.exs
Normal file
|
|
@ -0,0 +1,184 @@
|
||||||
|
defmodule Microwaveprop.Pskr.CalibrationSamplerTest do
|
||||||
|
use Microwaveprop.DataCase, async: true
|
||||||
|
|
||||||
|
alias Microwaveprop.Pskr.CalibrationSample
|
||||||
|
alias Microwaveprop.Pskr.CalibrationSampler
|
||||||
|
alias Microwaveprop.Pskr.SpotHourly
|
||||||
|
alias Microwaveprop.Repo
|
||||||
|
alias Microwaveprop.SpaceWeather.GeomagneticObservation
|
||||||
|
alias Microwaveprop.Weather.HrrrProfile
|
||||||
|
|
||||||
|
@hour ~U[2026-05-04 18:00:00Z]
|
||||||
|
|
||||||
|
defp insert_spot!(attrs) do
|
||||||
|
{:ok, _} =
|
||||||
|
%SpotHourly{}
|
||||||
|
|> SpotHourly.changeset(
|
||||||
|
Map.merge(
|
||||||
|
%{
|
||||||
|
hour_utc: @hour,
|
||||||
|
band: "10000",
|
||||||
|
sender_grid: "EM12KL",
|
||||||
|
receiver_grid: "DM43ST",
|
||||||
|
spot_count: 1,
|
||||||
|
midpoint_lat: 33.0,
|
||||||
|
midpoint_lon: -97.0,
|
||||||
|
distance_km: 200.0,
|
||||||
|
modes: ["FT8"]
|
||||||
|
},
|
||||||
|
attrs
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|> Repo.insert()
|
||||||
|
end
|
||||||
|
|
||||||
|
defp insert_hrrr!(attrs) do
|
||||||
|
{:ok, _} =
|
||||||
|
%HrrrProfile{}
|
||||||
|
|> HrrrProfile.changeset(
|
||||||
|
Map.merge(
|
||||||
|
%{
|
||||||
|
valid_time: @hour,
|
||||||
|
lat: 33.0,
|
||||||
|
lon: -97.0,
|
||||||
|
run_time: @hour,
|
||||||
|
surface_temp_c: 25.0,
|
||||||
|
surface_dewpoint_c: 18.0,
|
||||||
|
pwat_mm: 30.0,
|
||||||
|
surface_pressure_mb: 1013.0,
|
||||||
|
min_refractivity_gradient: -100.0,
|
||||||
|
hpbl_m: 500.0,
|
||||||
|
ducting_detected: false,
|
||||||
|
profile: []
|
||||||
|
},
|
||||||
|
attrs
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|> Repo.insert()
|
||||||
|
end
|
||||||
|
|
||||||
|
defp insert_kp!(value) do
|
||||||
|
{:ok, _} =
|
||||||
|
%GeomagneticObservation{}
|
||||||
|
|> GeomagneticObservation.changeset(%{
|
||||||
|
valid_time: DateTime.add(@hour, 60, :second),
|
||||||
|
kp_index: value
|
||||||
|
})
|
||||||
|
|> Repo.insert()
|
||||||
|
end
|
||||||
|
|
||||||
|
describe "build_for_hour/1" do
|
||||||
|
test "skips and returns 0 when no spots exist for the hour" do
|
||||||
|
assert CalibrationSampler.build_for_hour(@hour) == 0
|
||||||
|
assert Repo.aggregate(CalibrationSample, :count) == 0
|
||||||
|
end
|
||||||
|
|
||||||
|
test "builds one sample per (band, snapped midpoint) cell" do
|
||||||
|
insert_spot!(%{midpoint_lat: 33.01, midpoint_lon: -97.02})
|
||||||
|
insert_spot!(%{midpoint_lat: 33.04, midpoint_lon: -97.03, sender_grid: "EM13", spot_count: 3})
|
||||||
|
# Distinct grid cell (>0.125° away)
|
||||||
|
insert_spot!(%{midpoint_lat: 35.0, midpoint_lon: -97.0, sender_grid: "EM15"})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
|
||||||
|
assert CalibrationSampler.build_for_hour(@hour) == 2
|
||||||
|
[a, b] = CalibrationSample |> Repo.all() |> Enum.sort_by(& &1.midpoint_lat)
|
||||||
|
# Two near-midpoints collapsed; one distant cell stayed separate.
|
||||||
|
assert a.spot_count == 4
|
||||||
|
assert a.distinct_paths == 2
|
||||||
|
assert b.spot_count == 1
|
||||||
|
end
|
||||||
|
|
||||||
|
test "snaps midpoints to the 0.125° propagation grid" do
|
||||||
|
insert_spot!(%{midpoint_lat: 33.06, midpoint_lon: -97.07})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
# 33.06 → nearest 0.125° step → 33.0; -97.07 → -97.125
|
||||||
|
assert sample.midpoint_lat == 33.0
|
||||||
|
assert sample.midpoint_lon == -97.125
|
||||||
|
end
|
||||||
|
|
||||||
|
test "joins HRRR atmospheric features at the midpoint" do
|
||||||
|
insert_spot!(%{})
|
||||||
|
|
||||||
|
insert_hrrr!(%{
|
||||||
|
lat: 33.0,
|
||||||
|
lon: -97.0,
|
||||||
|
surface_temp_c: 22.5,
|
||||||
|
surface_dewpoint_c: 16.0,
|
||||||
|
pwat_mm: 28.0,
|
||||||
|
min_refractivity_gradient: -150.0,
|
||||||
|
hpbl_m: 420.0,
|
||||||
|
ducting_detected: true
|
||||||
|
})
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.surface_temp_c == 22.5
|
||||||
|
assert sample.surface_dewpoint_c == 16.0
|
||||||
|
assert sample.pwat_mm == 28.0
|
||||||
|
assert sample.min_refractivity_gradient == -150.0
|
||||||
|
assert sample.hpbl_m == 420.0
|
||||||
|
assert sample.hrrr_ducting_detected == true
|
||||||
|
end
|
||||||
|
|
||||||
|
test "leaves HRRR fields nil when no nearby profile exists" do
|
||||||
|
insert_spot!(%{})
|
||||||
|
# No HRRR profile inserted at all.
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.surface_temp_c == nil
|
||||||
|
assert sample.hpbl_m == nil
|
||||||
|
assert sample.hrrr_ducting_detected == nil
|
||||||
|
end
|
||||||
|
|
||||||
|
test "stamps the latest Kp from SpaceWeather on every sample in the hour" do
|
||||||
|
insert_spot!(%{})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
insert_kp!(5)
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.kp_index == 5
|
||||||
|
end
|
||||||
|
|
||||||
|
test "computes median distance across the cell's path mix" do
|
||||||
|
insert_spot!(%{distance_km: 100.0, sender_grid: "EM10"})
|
||||||
|
insert_spot!(%{distance_km: 200.0, sender_grid: "EM11"})
|
||||||
|
insert_spot!(%{distance_km: 300.0, sender_grid: "EM12"})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.median_distance_km == 200.0
|
||||||
|
end
|
||||||
|
|
||||||
|
test "deduplicates modes across the cell's spots" do
|
||||||
|
insert_spot!(%{modes: ["FT8"], sender_grid: "EM10"})
|
||||||
|
insert_spot!(%{modes: ["FT4", "FT8"], sender_grid: "EM11"})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert Enum.sort(sample.modes) == ["FT4", "FT8"]
|
||||||
|
end
|
||||||
|
|
||||||
|
test "is idempotent — re-running the same hour upserts the row" do
|
||||||
|
insert_spot!(%{})
|
||||||
|
insert_hrrr!(%{lat: 33.0, lon: -97.0})
|
||||||
|
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
first_count = Repo.aggregate(CalibrationSample, :count)
|
||||||
|
|
||||||
|
# Mutate the spot count and re-run; the sample must update, not duplicate.
|
||||||
|
Repo.update_all(SpotHourly, set: [spot_count: 99])
|
||||||
|
CalibrationSampler.build_for_hour(@hour)
|
||||||
|
second_count = Repo.aggregate(CalibrationSample, :count)
|
||||||
|
|
||||||
|
assert first_count == second_count
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.spot_count == 99
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
62
test/microwaveprop/workers/pskr_calibration_worker_test.exs
Normal file
62
test/microwaveprop/workers/pskr_calibration_worker_test.exs
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
defmodule Microwaveprop.Workers.PskrCalibrationWorkerTest do
|
||||||
|
use Microwaveprop.DataCase, async: true
|
||||||
|
use Oban.Testing, repo: Microwaveprop.Repo
|
||||||
|
|
||||||
|
alias Microwaveprop.Pskr.CalibrationSample
|
||||||
|
alias Microwaveprop.Pskr.SpotHourly
|
||||||
|
alias Microwaveprop.Repo
|
||||||
|
alias Microwaveprop.Workers.PskrCalibrationWorker
|
||||||
|
|
||||||
|
defp insert_spot!(hour_utc) do
|
||||||
|
{:ok, _} =
|
||||||
|
%SpotHourly{}
|
||||||
|
|> SpotHourly.changeset(%{
|
||||||
|
hour_utc: hour_utc,
|
||||||
|
band: "10000",
|
||||||
|
sender_grid: "EM12KL",
|
||||||
|
receiver_grid: "DM43ST",
|
||||||
|
spot_count: 1,
|
||||||
|
midpoint_lat: 33.0,
|
||||||
|
midpoint_lon: -97.0,
|
||||||
|
distance_km: 200.0,
|
||||||
|
modes: ["FT8"]
|
||||||
|
})
|
||||||
|
|> Repo.insert()
|
||||||
|
end
|
||||||
|
|
||||||
|
test "perform/1 with explicit hour_utc samples that hour" do
|
||||||
|
hour = ~U[2026-05-04 18:00:00Z]
|
||||||
|
insert_spot!(hour)
|
||||||
|
|
||||||
|
assert :ok =
|
||||||
|
perform_job(PskrCalibrationWorker, %{"hour_utc" => DateTime.to_iso8601(hour)})
|
||||||
|
|
||||||
|
[sample] = Repo.all(CalibrationSample)
|
||||||
|
assert sample.hour_utc == hour
|
||||||
|
end
|
||||||
|
|
||||||
|
test "perform/1 with no args defaults to the previous full hour" do
|
||||||
|
# Pick the hour just before "now" — the same logic the worker
|
||||||
|
# uses internally — so the spot we insert lines up with what the
|
||||||
|
# default arg path will pull.
|
||||||
|
target =
|
||||||
|
DateTime.utc_now()
|
||||||
|
|> DateTime.add(-3600, :second)
|
||||||
|
|> Map.put(:minute, 0)
|
||||||
|
|> Map.put(:second, 0)
|
||||||
|
|> Map.put(:microsecond, {0, 0})
|
||||||
|
|
||||||
|
insert_spot!(target)
|
||||||
|
assert :ok = perform_job(PskrCalibrationWorker, %{})
|
||||||
|
assert Repo.aggregate(CalibrationSample, :count) == 1
|
||||||
|
end
|
||||||
|
|
||||||
|
test "perform/1 returns :ok even when no spots exist for the hour" do
|
||||||
|
assert :ok =
|
||||||
|
perform_job(PskrCalibrationWorker, %{
|
||||||
|
"hour_utc" => "2026-05-04T03:00:00Z"
|
||||||
|
})
|
||||||
|
|
||||||
|
assert Repo.aggregate(CalibrationSample, :count) == 0
|
||||||
|
end
|
||||||
|
end
|
||||||
Loading…
Add table
Reference in a new issue