Graham McIntire
7a2b1f292c
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
64 lines
2.6 KiB
Elixir
64 lines
2.6 KiB
Elixir
defmodule Microwaveprop.Repo.Migrations.CreatePskrCalibrationSamples do
|
|
use Ecto.Migration
|
|
|
|
# Calibration corpus joining PSKR spot density (truth signal) with
|
|
# HRRR atmospheric state and SWPC space weather (predictors). One
|
|
# row per (hour, band, midpoint cell). The recalibrator reads from
|
|
# this table and computes predicted scores on demand from the
|
|
# stored features — never store the prediction itself, since it
|
|
# depends on the algorithm version under evaluation.
|
|
|
|
def change do
|
|
create table(:pskr_calibration_samples, primary_key: false) do
|
|
add :id, :binary_id, primary_key: true
|
|
add :hour_utc, :utc_datetime, null: false
|
|
add :band, :string, null: false
|
|
add :midpoint_lat, :float, null: false
|
|
add :midpoint_lon, :float, null: false
|
|
|
|
# Aggregate PSKR signal at this cell.
|
|
add :spot_count, :integer, null: false, default: 0
|
|
add :distinct_paths, :integer, null: false, default: 0
|
|
add :median_distance_km, :float
|
|
add :modes, {:array, :string}, null: false, default: []
|
|
|
|
# HRRR atmospheric snapshot at midpoint at the hour boundary.
|
|
# Nullable because the join may miss for pre-2014 cells or
|
|
# cells outside the HRRR domain.
|
|
add :surface_temp_c, :float
|
|
add :surface_dewpoint_c, :float
|
|
add :pwat_mm, :float
|
|
add :surface_pressure_mb, :float
|
|
add :min_refractivity_gradient, :float
|
|
add :hpbl_m, :float
|
|
add :hrrr_ducting_detected, :boolean
|
|
|
|
# SWPC geomagnetic state at the hour boundary. Same Kp value
|
|
# applies to every cell in the same hour, but stored per-row so
|
|
# the recalibrator doesn't have to re-join SpaceWeather for
|
|
# every batch of samples.
|
|
add :kp_index, :integer
|
|
|
|
timestamps(type: :utc_datetime)
|
|
end
|
|
|
|
# Same uniqueness shape as `pskr_spots_hourly` so the upsert
|
|
# pattern translates 1:1. Cell granularity is set by the sampler
|
|
# (currently 0.125° to match the propagation grid).
|
|
create unique_index(:pskr_calibration_samples, [
|
|
:hour_utc,
|
|
:band,
|
|
:midpoint_lat,
|
|
:midpoint_lon
|
|
])
|
|
|
|
create index(:pskr_calibration_samples, [:hour_utc])
|
|
|
|
# Make the spot-side spatial filter the sampler runs cheap.
|
|
# `pskr_spots_hourly` already has unique-index coverage on
|
|
# (hour_utc, band, sender_grid, receiver_grid) but no index that
|
|
# supports a midpoint range scan, which is the hot path for both
|
|
# the sampler join and any future map-side density queries.
|
|
create index(:pskr_spots_hourly, [:midpoint_lat, :midpoint_lon])
|
|
end
|
|
end
|