defmodule Microwaveprop.Pskr do @moduledoc """ Ingestion of the PSK Reporter MQTT firehose (`mqtt.pskreporter.info:1883`). ## Why we listen Each MQTT message is a single reception report — *station X heard station Y at SNR Z on band B at time T*. Read across millions of reports, this is a continuous, geolocated, timestamped record of *did propagation actually occur on this path right now?* That's the empirical signal we correlate against HRRR atmospheric state to recalibrate the propagation scoring weights. ## What we keep Topic structure (from the broker landing page): pskr/filter/v2//////// We subscribe with country-side wildcards anchored on USA (DXCC 291), so the broker pre-filters to spots involving at least one CONUS station — anything else can't be cross-referenced with HRRR anyway. Per-spot payload (verbatim keys, also documented at http://mqtt.pskreporter.info/): sq f md rp t t_tx sc sl rc rl sa ra b ## What we discard The aggregator collapses every spot inside `(hour_utc, band, sender_grid, receiver_grid)` into one `pskr_spots_hourly` row. We keep counts + SNR envelope + the set of modes — the individual report identity, sequence number, and exact intra-hour timestamp are dropped on the floor. A 50-reporter QSO becomes a single bumped count, not 50 rows; that's the design, not a limitation. Locator precision is preserved as reported: a spotter who shares `EM12kl` keeps subsquare resolution, while a `EM12kl37` reporter keeps extended-square resolution. The bucket key uses the full grid, so two spotters in different subsquares of the same field produce two rows — that's the right behavior, since they're literally different paths. """ alias Microwaveprop.Geo alias Microwaveprop.Radio.Maidenhead require Logger @typedoc "Parsed PSK Reporter spot, post-`parse_spot/1`." @type spot :: %{ band: String.t(), mode: String.t(), frequency_hz: pos_integer(), snr_db: integer(), transmit_time: DateTime.t(), sender_grid: String.t(), receiver_grid: String.t(), sender_callsign: String.t() | nil, receiver_callsign: String.t() | nil, sender_country: integer() | nil, receiver_country: integer() | nil } @typedoc "Bucket key used by the aggregator's accumulator map." @type path_key :: {DateTime.t(), String.t(), String.t(), String.t()} @doc """ Decodes a single MQTT payload (binary JSON) into a normalized spot map. Locators are kept at the precision the spotter reported (4/6/8/10 chars) and uppercased for consistent storage. Returns `{:error, reason}` for any payload missing the fields we need — the caller should count and discard. Callsigns are taken from the JSON payload (`sc`/`rc` keys). When those keys are absent or empty (common for FT8/Q65 multi- decoder spots), falls back to callsigns extracted from the MQTT topic which always carries them in the routing path. """ @spec parse_spot(binary(), keyword()) :: {:ok, spot()} | {:error, term()} def parse_spot(payload, opts \\ []) when is_binary(payload) do with {:ok, json} <- Jason.decode(payload), {:ok, sender_grid} <- fetch_grid(json, "sl"), {:ok, receiver_grid} <- fetch_grid(json, "rl"), {:ok, band} <- fetch_string(json, "b"), {:ok, time_unix} <- fetch_time(json), {:ok, transmit_time} <- DateTime.from_unix(time_unix) do {:ok, %{ band: band, mode: Map.get(json, "md", ""), frequency_hz: Map.get(json, "f"), snr_db: Map.get(json, "rp"), transmit_time: transmit_time, sender_grid: sender_grid, receiver_grid: receiver_grid, sender_callsign: normalize_callsign(Map.get(json, "sc")) || normalize_callsign(Keyword.get(opts, :sender_callsign)), receiver_callsign: normalize_callsign(Map.get(json, "rc")) || normalize_callsign(Keyword.get(opts, :receiver_callsign)), sender_country: Map.get(json, "sa"), receiver_country: Map.get(json, "ra") }} end end @doc "Bucket a spot to (top-of-hour, band, sender_grid, receiver_grid) at full reported precision." @spec path_key(spot()) :: path_key() def path_key(%{transmit_time: t, band: band, sender_grid: snd, receiver_grid: rcv}) do {%{t | minute: 0, second: 0, microsecond: {0, 0}}, band, snd, rcv} end @doc """ Fold a spot into an accumulator row. Pass `nil` as the first argument to seed a fresh row. The result is an attribute map ready to feed into `Repo.insert_all/3` / `Ecto.Changeset.cast/3`. Pre-computes sender/receiver lat-lon, midpoint, and great-circle distance once at first sight — they don't change for subsequent spots on the same path, so the merge clause is cheap. """ @spec merge_spot(map() | nil, spot()) :: map() def merge_spot(nil, spot) do {hour, band, snd, rcv} = path_key(spot) {snd_lat, snd_lon} = grid_center(snd) {rcv_lat, rcv_lon} = grid_center(rcv) distance = Geo.haversine_km(snd_lat, snd_lon, rcv_lat, rcv_lon) %{ hour_utc: hour, band: band, sender_grid: snd, receiver_grid: rcv, sender_country: spot.sender_country, receiver_country: spot.receiver_country, sender_lat: snd_lat, sender_lon: snd_lon, receiver_lat: rcv_lat, receiver_lon: rcv_lon, midpoint_lat: (snd_lat + rcv_lat) / 2, midpoint_lon: (snd_lon + rcv_lon) / 2, distance_km: distance, spot_count: 1, max_snr_db: spot.snr_db, min_snr_db: spot.snr_db, modes: maybe_mode_list(spot.mode), sender_callsigns: maybe_callsign_list(spot.sender_callsign), receiver_callsigns: maybe_callsign_list(spot.receiver_callsign), first_spot_at: spot.transmit_time, last_spot_at: spot.transmit_time } end def merge_spot(%{} = acc, spot) do %{ acc | spot_count: acc.spot_count + 1, max_snr_db: max_snr(acc.max_snr_db, spot.snr_db), min_snr_db: min_snr(acc.min_snr_db, spot.snr_db), modes: add_mode(acc.modes, spot.mode), sender_callsigns: add_callsign(acc.sender_callsigns, spot.sender_callsign), receiver_callsigns: add_callsign(acc.receiver_callsigns, spot.receiver_callsign), first_spot_at: earliest(acc.first_spot_at, spot.transmit_time), last_spot_at: latest(acc.last_spot_at, spot.transmit_time) } end # Require ≥ 4-char locators on both ends. 4-char grids cover # ~70 km × 100 km — less precise for HRRR calibration but still # useful for spot display and coarse path analysis. # # Maidenhead validity enforces even length + correct alphabet # per position, so a passing string is a legal 4/6/8/10-char # locator we can store as-is. Uppercase for consistent indexing # — the unique key is case-sensitive. defp fetch_grid(json, key) do case Map.get(json, key) do grid when is_binary(grid) -> upper = String.upcase(grid) cond do not Maidenhead.valid?(upper) -> {:error, {:bad_grid, key, grid}} String.length(upper) < 4 -> {:error, {:short_grid, key, grid}} true -> {:ok, upper} end other -> {:error, {:missing_grid, key, other}} end end defp fetch_string(json, key) do case Map.get(json, key) do v when is_binary(v) and v != "" -> {:ok, v} other -> {:error, {:missing, key, other}} end end # `t_tx` is the transmission time and is the right anchor for # hour-bucketing — that's when propagation actually carried the # signal. `t` (PSKReporter receive time) drifts a few seconds and # would split spots that crossed an hour boundary onto two rows. defp fetch_time(json) do case {Map.get(json, "t_tx"), Map.get(json, "t")} do {n, _} when is_integer(n) -> {:ok, n} {_, n} when is_integer(n) -> {:ok, n} _ -> {:error, :no_timestamp} end end defp grid_center(grid) do case Geo.maidenhead_center(grid) do nil -> Logger.warning("Pskr.grid_center: Geo.maidenhead_center returned nil for grid=#{grid}, using {0.0, 0.0}") {0.0, 0.0} center -> center end end defp maybe_mode_list(""), do: [] defp maybe_mode_list(mode), do: [mode] defp add_mode(modes, ""), do: modes defp add_mode(modes, nil), do: modes defp add_mode(modes, mode), do: if(mode in modes, do: modes, else: [mode | modes]) defp normalize_callsign(nil), do: nil defp normalize_callsign(""), do: nil defp normalize_callsign(call), do: String.upcase(call) defp maybe_callsign_list(nil), do: [] defp maybe_callsign_list(call), do: [call] defp add_callsign(calls, nil), do: calls defp add_callsign(calls, call), do: if(call in calls, do: calls, else: [call | calls]) defp max_snr(nil, b), do: b defp max_snr(a, nil), do: a defp max_snr(a, b), do: max(a, b) defp min_snr(nil, b), do: b defp min_snr(a, nil), do: a defp min_snr(a, b), do: min(a, b) defp earliest(a, b), do: if(DateTime.before?(a, b), do: a, else: b) defp latest(a, b), do: if(DateTime.after?(a, b), do: a, else: b) end