Graham McIntire
7a2b1f292c
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
80 lines
2.8 KiB
Elixir
80 lines
2.8 KiB
Elixir
defmodule Microwaveprop.Pskr.CalibrationSample do
|
|
@moduledoc """
|
|
One sample for the PSKR-driven recalibration corpus: a (hour, band,
|
|
midpoint cell) bucket joining aggregate PSKR spot density with the
|
|
HRRR atmospheric state and SWPC geomagnetic state at that hour.
|
|
|
|
## Why we don't store predicted scores
|
|
|
|
The corpus exists so the recalibrator can learn weights from
|
|
(atmospheric features → observed spot density). Predicted scores
|
|
are derived quantities that change every time the algorithm
|
|
changes; storing them locks the corpus to a single algorithm
|
|
version. Features are immutable — they describe what the
|
|
atmosphere actually was — so the same sample row stays useful
|
|
across every weight refit.
|
|
|
|
## Cell granularity
|
|
|
|
`midpoint_lat`/`midpoint_lon` are snapped to the propagation grid
|
|
(currently 0.125°) by `Pskr.CalibrationSampler` before insert.
|
|
Two PSKR paths whose midpoints land in the same grid cell collapse
|
|
into a single row — the same logic that already drives
|
|
`pskr_spots_hourly` aggregation, applied at the spatial dimension.
|
|
|
|
## Forward-only
|
|
|
|
PSK Reporter has no historical archive — the firehose is real-time
|
|
only. The corpus grows from when the feed first started and never
|
|
backfills further. Recalibration weight refits should only be
|
|
attempted once enough days have accumulated to cover diurnal and
|
|
synoptic variability (heuristic: ≥ 30 d, ≥ 10k samples).
|
|
"""
|
|
use Ecto.Schema
|
|
|
|
import Ecto.Changeset
|
|
|
|
@primary_key {:id, :binary_id, autogenerate: true}
|
|
@foreign_key_type :binary_id
|
|
|
|
schema "pskr_calibration_samples" do
|
|
field :hour_utc, :utc_datetime
|
|
field :band, :string
|
|
field :midpoint_lat, :float
|
|
field :midpoint_lon, :float
|
|
|
|
field :spot_count, :integer, default: 0
|
|
field :distinct_paths, :integer, default: 0
|
|
field :median_distance_km, :float
|
|
field :modes, {:array, :string}, default: []
|
|
|
|
field :surface_temp_c, :float
|
|
field :surface_dewpoint_c, :float
|
|
field :pwat_mm, :float
|
|
field :surface_pressure_mb, :float
|
|
field :min_refractivity_gradient, :float
|
|
field :hpbl_m, :float
|
|
field :hrrr_ducting_detected, :boolean
|
|
|
|
field :kp_index, :integer
|
|
|
|
timestamps(type: :utc_datetime)
|
|
end
|
|
|
|
@type t :: %__MODULE__{}
|
|
|
|
@cast_fields ~w(hour_utc band midpoint_lat midpoint_lon spot_count distinct_paths
|
|
median_distance_km modes surface_temp_c surface_dewpoint_c pwat_mm
|
|
surface_pressure_mb min_refractivity_gradient hpbl_m hrrr_ducting_detected
|
|
kp_index)a
|
|
|
|
@required_fields ~w(hour_utc band midpoint_lat midpoint_lon spot_count)a
|
|
|
|
@spec changeset(t(), map()) :: Ecto.Changeset.t()
|
|
def changeset(record, attrs) do
|
|
record
|
|
|> cast(attrs, @cast_fields)
|
|
|> validate_required(@required_fields)
|
|
|> unique_constraint([:hour_utc, :band, :midpoint_lat, :midpoint_lon])
|
|
end
|
|
end
|