Graham McIntire
7a2b1f292c
PSK Reporter is now an always-on data feed, so the calibration
corpus that recalibration will eventually train against can grow
hourly from when the firehose started. One row per (hour, band,
0.125° midpoint cell) joins three sources:
* `pskr_spots_hourly` — observed spot density (truth signal)
* `hrrr_profiles` nearest match at the cell midpoint at hour
boundary (atmospheric features: T, Td, PWAT, P, dN/dh, HPBL,
ducting flag)
* `geomagnetic_observations` latest Kp at the hour (space weather)
Predicted scores are intentionally NOT stored — they're a function
of the algorithm version under evaluation. Storing only features
keeps the corpus stable across every weight refit; the recalibrator
computes predictions on demand.
Components:
* Migration `20260504210756_create_pskr_calibration_samples` —
new table + unique index on (hour, band, midpoint_lat, lon),
plus a midpoint spatial index on `pskr_spots_hourly` to keep
the join cheap.
* `Pskr.CalibrationSample` — schema mirror of the table with
the same `(hour, band, midpoint_lat, lon)` unique constraint.
* `Pskr.CalibrationSampler.build_for_hour/1` — pulls all spots
for the hour, snaps midpoints to the propagation grid (0.125°),
builds an in-memory HRRR index over a ±0.07°/±60 min window,
and bulk-upserts samples. Idempotent — re-runs upsert.
* `Workers.PskrCalibrationWorker` — Oban cron entry on the
`:backfill_enqueue` queue. Default args target the previous
full hour; explicit `hour_utc` arg reruns any hour.
* Cron `25 * * * *` — fires past HRRR analysis publish window
(~HH:50→HH:05) and PSKR aggregator's 60s flush.
Forward-only: PSKR has no historical archive, so the corpus only
grows from when the feed started. Recalibration weight refits
should wait for ~30 days / ~10k samples to cover diurnal and
synoptic variability.
Tests cover cell-snapping, HRRR feature joining, Kp stamping,
median-distance aggregation, mode dedup, idempotent reruns, and
the worker's default-hour and explicit-hour args (12 new tests,
all passing).
Backfill pipeline untouched — none of these changes feed into
contact enrichment.
62 lines
1.8 KiB
Elixir
62 lines
1.8 KiB
Elixir
defmodule Microwaveprop.Workers.PskrCalibrationWorkerTest do
|
|
use Microwaveprop.DataCase, async: true
|
|
use Oban.Testing, repo: Microwaveprop.Repo
|
|
|
|
alias Microwaveprop.Pskr.CalibrationSample
|
|
alias Microwaveprop.Pskr.SpotHourly
|
|
alias Microwaveprop.Repo
|
|
alias Microwaveprop.Workers.PskrCalibrationWorker
|
|
|
|
defp insert_spot!(hour_utc) do
|
|
{:ok, _} =
|
|
%SpotHourly{}
|
|
|> SpotHourly.changeset(%{
|
|
hour_utc: hour_utc,
|
|
band: "10000",
|
|
sender_grid: "EM12KL",
|
|
receiver_grid: "DM43ST",
|
|
spot_count: 1,
|
|
midpoint_lat: 33.0,
|
|
midpoint_lon: -97.0,
|
|
distance_km: 200.0,
|
|
modes: ["FT8"]
|
|
})
|
|
|> Repo.insert()
|
|
end
|
|
|
|
test "perform/1 with explicit hour_utc samples that hour" do
|
|
hour = ~U[2026-05-04 18:00:00Z]
|
|
insert_spot!(hour)
|
|
|
|
assert :ok =
|
|
perform_job(PskrCalibrationWorker, %{"hour_utc" => DateTime.to_iso8601(hour)})
|
|
|
|
[sample] = Repo.all(CalibrationSample)
|
|
assert sample.hour_utc == hour
|
|
end
|
|
|
|
test "perform/1 with no args defaults to the previous full hour" do
|
|
# Pick the hour just before "now" — the same logic the worker
|
|
# uses internally — so the spot we insert lines up with what the
|
|
# default arg path will pull.
|
|
target =
|
|
DateTime.utc_now()
|
|
|> DateTime.add(-3600, :second)
|
|
|> Map.put(:minute, 0)
|
|
|> Map.put(:second, 0)
|
|
|> Map.put(:microsecond, {0, 0})
|
|
|
|
insert_spot!(target)
|
|
assert :ok = perform_job(PskrCalibrationWorker, %{})
|
|
assert Repo.aggregate(CalibrationSample, :count) == 1
|
|
end
|
|
|
|
test "perform/1 returns :ok even when no spots exist for the hour" do
|
|
assert :ok =
|
|
perform_job(PskrCalibrationWorker, %{
|
|
"hour_utc" => "2026-05-04T03:00:00Z"
|
|
})
|
|
|
|
assert Repo.aggregate(CalibrationSample, :count) == 0
|
|
end
|
|
end
|