HRRR forecast hours (f00-f18) with timeline map UI
- HrrrClient.hrrr_url accepts forecast_hour param (wrfsfcfHH.grib2) - PropagationGridWorker fetches all 19 forecast hours per run - Propagation.scores_at/3 queries scores at specific valid_time - Propagation.available_valid_times/1 returns all forecast times for timeline - Pruning keeps scores with valid_time >= now - 2h (forecast-aware) - MapLive: select_time event, timeline data pushed to JS - JS: forecast timeline bar at bottom of map with clickable hour buttons - PubSub broadcast sends list of valid_times instead of single time
This commit is contained in:
parent
c775eb2611
commit
e82e631135
8 changed files with 558 additions and 86 deletions
|
|
@ -450,6 +450,26 @@ export const PropagationMap = {
|
|||
})
|
||||
|
||||
|
||||
// --- Forecast timeline ---
|
||||
this.timelineEl = document.getElementById("forecast-timeline")
|
||||
this.timelineData = JSON.parse(this.el.dataset.validTimes || "[]")
|
||||
this.selectedTime = this.el.dataset.selectedTime || null
|
||||
if (this.timelineData.length > 1) {
|
||||
this.renderTimeline()
|
||||
}
|
||||
|
||||
this.handleEvent("update_timeline", ({ times, selected }) => {
|
||||
this.timelineData = times
|
||||
this.selectedTime = selected
|
||||
this.renderTimeline()
|
||||
})
|
||||
|
||||
// Prevent timeline from eating map clicks
|
||||
if (this.timelineEl) {
|
||||
L.DomEvent.disableClickPropagation(this.timelineEl)
|
||||
L.DomEvent.disableScrollPropagation(this.timelineEl)
|
||||
}
|
||||
|
||||
this.el.addEventListener("show-loading", () => topbar.show(300))
|
||||
|
||||
this.sendBounds()
|
||||
|
|
@ -604,6 +624,67 @@ export const PropagationMap = {
|
|||
return { ...data, ...this.bandInfo }
|
||||
},
|
||||
|
||||
renderTimeline() {
|
||||
if (!this.timelineEl || this.timelineData.length <= 1) {
|
||||
if (this.timelineEl) this.timelineEl.style.display = "none"
|
||||
return
|
||||
}
|
||||
|
||||
const now = new Date()
|
||||
const items = this.timelineData.map(t => {
|
||||
const dt = new Date(t.time)
|
||||
const isSelected = t.time === this.selectedTime
|
||||
const isPast = dt < now
|
||||
const isNow = Math.abs(dt - now) < 3600000 // within 1 hour
|
||||
const offsetH = Math.round((dt - now) / 3600000)
|
||||
const label = isNow ? "Now" : (offsetH > 0 ? `+${offsetH}h` : `${offsetH}h`)
|
||||
return { ...t, dt, isSelected, isPast, isNow, label, offsetH }
|
||||
})
|
||||
|
||||
// Find trend: compare first (now) vs a few hours ahead
|
||||
const nowItem = items.find(i => i.isNow) || items[0]
|
||||
const futureItem = items.find(i => i.offsetH >= 3) || items[items.length - 1]
|
||||
|
||||
let trendHint = ""
|
||||
if (nowItem && futureItem && this.gridLookup) {
|
||||
// Sample center of viewport
|
||||
const center = this.map.getCenter()
|
||||
const nowScore = this.lookupPoint(center.lat, center.lng)
|
||||
// Can't look up future scores client-side, so just show the time range
|
||||
trendHint = ""
|
||||
}
|
||||
|
||||
const buttons = items.map(t => {
|
||||
const bg = t.isSelected ? "#00ffa3" : (t.isPast ? "rgba(255,255,255,0.08)" : "rgba(255,255,255,0.15)")
|
||||
const fg = t.isSelected ? "#000" : (t.isPast ? "rgba(255,255,255,0.4)" : "#fff")
|
||||
const border = t.isSelected ? "2px solid #00ffa3" : "1px solid rgba(255,255,255,0.2)"
|
||||
const weight = t.isSelected || t.isNow ? "700" : "400"
|
||||
return `<button data-time="${t.time}" style="
|
||||
padding:4px 8px;font-size:11px;font-weight:${weight};
|
||||
background:${bg};color:${fg};border:${border};border-radius:6px;
|
||||
cursor:pointer;white-space:nowrap;min-width:38px;
|
||||
transition:background 0.15s;
|
||||
">${t.label}<br><span style="font-size:9px;opacity:0.7;">${t.dt.getUTCHours().toString().padStart(2, "0")}:00</span></button>`
|
||||
}).join("")
|
||||
|
||||
this.timelineEl.style.display = "block"
|
||||
this.timelineEl.innerHTML = `
|
||||
<div style="background:rgba(0,0,0,0.85);border-radius:12px;padding:6px 10px;display:flex;gap:3px;align-items:center;box-shadow:0 2px 12px rgba(0,0,0,0.4);overflow-x:auto;max-width:90vw;">
|
||||
<span style="font-size:10px;color:rgba(255,255,255,0.5);margin-right:4px;white-space:nowrap;">Forecast</span>
|
||||
${buttons}
|
||||
</div>`
|
||||
|
||||
// Attach click handlers
|
||||
this.timelineEl.querySelectorAll("button[data-time]").forEach(btn => {
|
||||
btn.addEventListener("click", () => {
|
||||
this.selectedTime = btn.dataset.time
|
||||
this.renderTimeline()
|
||||
topbar.show()
|
||||
this.pushEvent("select_time", { time: btn.dataset.time })
|
||||
})
|
||||
})
|
||||
},
|
||||
|
||||
sendBounds() {
|
||||
topbar.show()
|
||||
const b = this.map.getBounds()
|
||||
|
|
|
|||
81
docs/goes-r-research.md
Normal file
81
docs/goes-r-research.md
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
# GOES-R Satellite Data for Propagation Prediction
|
||||
|
||||
## Summary
|
||||
|
||||
**Verdict: Low priority. GOES-R data is already baked into HRRR, and the satellite cannot resolve the vertical gradients that cause ducting.**
|
||||
|
||||
## What is "GOES-R Rapid Refresh"?
|
||||
|
||||
There is no product called "GOES-R Rapid Refresh" or "GRR." These are two separate systems:
|
||||
- **GOES-R** — geostationary satellite series (GOES-16, 17, 18, 19) with the Advanced Baseline Imager (ABI)
|
||||
- **HRRR** — High-Resolution Rapid Refresh, a ground-based NWP model
|
||||
|
||||
They are complementary: NOAA assimilates GOES-R satellite observations (cloud-top data, satellite winds) into the HRRR model. By consuming HRRR output, we are already indirectly consuming GOES-R data — processed through full 3D physics.
|
||||
|
||||
## Relevant GOES-R Products
|
||||
|
||||
| Product | Resolution | Cadence | Clear Sky Only? | Relevance |
|
||||
|---------|-----------|---------|-----------------|-----------|
|
||||
| Derived Stability Indices (K-Index, LI, CAPE) | 10 km | 5 min | Yes | Moderate — but K-index inversely correlated with ducting per our analysis |
|
||||
| Total Precipitable Water | 10 km | 5 min | Yes | Low — our analysis found PWAT is NOT a ducting discriminator |
|
||||
| Vertical Temperature Profile | 10 km | 5 min | Yes | **Very low** — see critical caveat below |
|
||||
| Vertical Moisture Profile | 10 km | 5 min | Yes | Same limitation as temperature |
|
||||
| Land Surface Temperature | 2 km | 5 min | Yes | Minor — land-sea contrasts drive coastal ducting |
|
||||
| Sea Surface Temperature | 2 km | hourly | Yes | Same — coastal ducting indicator |
|
||||
| Rain Rate | 2 km | 5 min | No | Redundant — HRRR already provides precip |
|
||||
| Derived Motion Winds | varies | 5 min | Partially | Redundant — HRRR already provides wind fields |
|
||||
| Cloud Imagery (water vapor) | 2 km | 1-5 min | No | Visual supplement only — moisture plume tracking |
|
||||
|
||||
## Critical Limitation: Cannot Resolve Ducting
|
||||
|
||||
The ABI is an imager with 16 spectral bands, not a hyperspectral sounder. Key problems:
|
||||
|
||||
1. **The "vertical profiles" lean heavily on the NWP model first guess.** The ABI lacks the spectral resolution to independently resolve temperature/moisture structure. The 54-level temperature profile sounds impressive, but with only 16 IR bands, the independent information content is very low. The retrieved profiles "retain features of the first guess" — meaning they're essentially a slightly nudged version of the same NWP model we're already using.
|
||||
|
||||
2. **Ducting requires detecting gradients over 50-200m vertical scales.** The ABI simply cannot resolve this. A 100m-thick temperature inversion looks identical to a smooth profile from the satellite's perspective.
|
||||
|
||||
3. **Clear-sky only.** All the useful derived products (TPW, stability indices, profiles) only work under cloud-free conditions. HRRR gives us data everywhere, always.
|
||||
|
||||
## Comparison: What GOES-R Adds vs. What We Have
|
||||
|
||||
| What We Need | HRRR (current) | GOES-R (proposed) |
|
||||
|---|---|---|
|
||||
| Surface temp/dewpoint | 3 km, hourly, all-sky | 10 km, 5 min, clear-sky only |
|
||||
| Vertical profiles | 8 pressure levels with T/Td/height, all-sky | 54 levels but low information content, clear-sky only |
|
||||
| Refractivity gradient | Computed from HRRR profiles — our key ducting indicator | Cannot resolve the vertical structure needed |
|
||||
| Boundary layer depth | HPBL from model physics | No direct product |
|
||||
| Wind | 10m wind components, all-sky | Cloud-tracked winds at various levels |
|
||||
| Precipitation | Model physics + radar assimilation | IR-only rain rate (less accurate) |
|
||||
| Forecast capability | 18-48 hours ahead | None — observation only |
|
||||
|
||||
## Where GOES-R Could Help (Minor)
|
||||
|
||||
**Real-time nowcasting between HRRR cycles:** HRRR updates hourly with ~60 min latency, meaning displayed data can be up to 2+ hours old. GOES-R's 5-minute cadence could fill the gap for:
|
||||
- Cloud cover changes (water vapor imagery)
|
||||
- Rapid destabilization (stability indices flagging convective onset)
|
||||
- Land-sea temperature contrasts (coastal ducting indicator)
|
||||
|
||||
These would be supplementary monitoring products, not improvements to the core scoring algorithm.
|
||||
|
||||
## Data Access
|
||||
|
||||
All GOES-R data is freely available on AWS S3 (no auth):
|
||||
- `s3://noaa-goes19/` (current GOES-East, operational since April 2025)
|
||||
- `s3://noaa-goes18/` (current GOES-West)
|
||||
- Path: `ABI-L2-{PRODUCT}{REGION}/{YEAR}/{DOY}/{HOUR}/{filename}.nc`
|
||||
- Format: NetCDF4
|
||||
- Also available on Microsoft Planetary Computer (Azure)
|
||||
|
||||
## Future: GeoXO Hyperspectral Sounder
|
||||
|
||||
The upcoming GeoXO satellite series (late 2020s/early 2030s) will carry a true hyperspectral IR sounder (GXS) with ~1,550 spectral bands. This would dramatically improve vertical temperature/moisture profiling from geostationary orbit and could genuinely detect inversion layers relevant to ducting. Worth revisiting when GeoXO data becomes available.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Don't integrate GOES-R now.** The HRRR-based approach already consumes better data for ducting prediction than GOES-R can provide directly. The complexity cost (S3 polling, NetCDF parsing, spatial interpolation, handling clear-sky gaps) is not justified by the marginal benefit.
|
||||
|
||||
**Priority order for improving predictions:**
|
||||
1. HRRR forecast hours (f01-f18) — already designed in `prediction.md`
|
||||
2. Radiosonde data (already partially ingested — better vertical profiles than any satellite)
|
||||
3. GeoXO hyperspectral sounder (when available, late 2020s)
|
||||
4. GOES-R products (low value-add given HRRR already assimilates them)
|
||||
|
|
@ -57,8 +57,13 @@ defmodule Microwaveprop.Propagation do
|
|||
end)
|
||||
end
|
||||
|
||||
@doc "Upsert propagation scores in batches within a transaction so readers see all-or-nothing."
|
||||
def upsert_scores(scores) do
|
||||
@doc """
|
||||
Upsert propagation scores in batches within a transaction so readers see all-or-nothing.
|
||||
|
||||
Options:
|
||||
* `:prune` - whether to prune old scores after upsert (default true)
|
||||
"""
|
||||
def upsert_scores(scores, opts \\ []) do
|
||||
now = DateTime.truncate(DateTime.utc_now(), :second)
|
||||
|
||||
entries =
|
||||
|
|
@ -94,42 +99,55 @@ defmodule Microwaveprop.Propagation do
|
|||
timeout: :infinity
|
||||
)
|
||||
|
||||
case result do
|
||||
{:ok, _count} -> prune_old_scores()
|
||||
_ -> :ok
|
||||
if Keyword.get(opts, :prune, true) do
|
||||
case result do
|
||||
{:ok, _count} -> prune_old_scores()
|
||||
_ -> :ok
|
||||
end
|
||||
end
|
||||
|
||||
result
|
||||
end
|
||||
|
||||
defp prune_old_scores do
|
||||
keep_times =
|
||||
Repo.all(
|
||||
from(gs in GridScore, select: gs.valid_time, distinct: gs.valid_time, order_by: [desc: gs.valid_time], limit: 2)
|
||||
)
|
||||
@doc "Remove scores with valid_times older than 1 hour before the earliest current forecast."
|
||||
def prune_old_scores do
|
||||
# Keep all forecast hours from the latest run + 1 hour buffer for the previous run
|
||||
cutoff = DateTime.add(DateTime.utc_now(), -2, :hour)
|
||||
|
||||
if length(keep_times) == 2 do
|
||||
oldest_kept = List.last(keep_times)
|
||||
{deleted, _} = Repo.delete_all(from(gs in GridScore, where: gs.valid_time < ^cutoff))
|
||||
|
||||
{deleted, _} = Repo.delete_all(from(gs in GridScore, where: gs.valid_time < ^oldest_kept))
|
||||
|
||||
if deleted > 0 do
|
||||
Logger.info("PropagationScores: pruned #{deleted} old scores, keeping 2 most recent valid_times")
|
||||
end
|
||||
if deleted > 0 do
|
||||
Logger.info("PropagationScores: pruned #{deleted} old scores (valid_time < #{cutoff})")
|
||||
end
|
||||
end
|
||||
|
||||
@doc "Get the latest scores for a band, optionally within a bounding box. Excludes factors for performance."
|
||||
def latest_scores(band_mhz, bounds \\ nil) do
|
||||
latest_time = latest_valid_time(band_mhz)
|
||||
@doc "Returns all distinct valid_times for a band, ordered ascending."
|
||||
def available_valid_times(band_mhz) do
|
||||
Repo.all(
|
||||
from(gs in GridScore,
|
||||
where: gs.band_mhz == ^band_mhz,
|
||||
select: gs.valid_time,
|
||||
distinct: gs.valid_time,
|
||||
order_by: [asc: gs.valid_time]
|
||||
)
|
||||
)
|
||||
end
|
||||
|
||||
case latest_time do
|
||||
@doc """
|
||||
Get scores for a band at a specific valid_time, optionally within a bounding box.
|
||||
If valid_time is nil, uses the earliest available (current analysis hour).
|
||||
Excludes factors for performance.
|
||||
"""
|
||||
def scores_at(band_mhz, valid_time, bounds \\ nil) do
|
||||
time = valid_time || earliest_valid_time(band_mhz)
|
||||
|
||||
case time do
|
||||
nil ->
|
||||
[]
|
||||
|
||||
_ ->
|
||||
from(gs in GridScore,
|
||||
where: gs.band_mhz == ^band_mhz and gs.valid_time == ^latest_time,
|
||||
where: gs.band_mhz == ^band_mhz and gs.valid_time == ^time,
|
||||
select: %{lat: gs.lat, lon: gs.lon, score: gs.score, valid_time: gs.valid_time}
|
||||
)
|
||||
|> maybe_filter_bounds(bounds)
|
||||
|
|
@ -137,6 +155,15 @@ defmodule Microwaveprop.Propagation do
|
|||
end
|
||||
end
|
||||
|
||||
@doc "Get the latest scores for a band (alias for scores_at with earliest valid_time)."
|
||||
def latest_scores(band_mhz, bounds \\ nil) do
|
||||
scores_at(band_mhz, nil, bounds)
|
||||
end
|
||||
|
||||
defp earliest_valid_time(band_mhz) do
|
||||
Repo.one(from(gs in GridScore, where: gs.band_mhz == ^band_mhz, select: min(gs.valid_time)))
|
||||
end
|
||||
|
||||
@doc "Get the full score and factors for a specific grid point, snapped to nearest grid."
|
||||
def point_detail(band_mhz, lat, lon) do
|
||||
step = Grid.step()
|
||||
|
|
|
|||
|
|
@ -27,15 +27,16 @@ defmodule Microwaveprop.Weather.HrrrClient do
|
|||
|
||||
def surface_messages, do: @surface_messages
|
||||
|
||||
def fetch_grid(points, valid_time, opts \\ []) do
|
||||
hour_dt = nearest_hrrr_hour(valid_time)
|
||||
def fetch_grid(points, run_time, opts \\ []) do
|
||||
hour_dt = nearest_hrrr_hour(run_time)
|
||||
date = DateTime.to_date(hour_dt)
|
||||
hour = hour_dt.hour
|
||||
forecast_hour = Keyword.get(opts, :forecast_hour, 0)
|
||||
|
||||
with {:ok, sfc_grid} <- fetch_product_grid(date, hour, :surface, points) do
|
||||
with {:ok, sfc_grid} <- fetch_product_grid(date, hour, :surface, points, forecast_hour) do
|
||||
# Pressure is optional — if it fails, continue with surface-only data
|
||||
prs_grid =
|
||||
case maybe_fetch_pressure_grid(date, hour, points, opts) do
|
||||
case maybe_fetch_pressure_grid(date, hour, points, opts, forecast_hour) do
|
||||
{:ok, grid} ->
|
||||
grid
|
||||
|
||||
|
|
@ -93,14 +94,15 @@ defmodule Microwaveprop.Weather.HrrrClient do
|
|||
end
|
||||
end
|
||||
|
||||
def hrrr_url(date, hour, product) do
|
||||
def hrrr_url(date, hour, product, forecast_hour \\ 0) do
|
||||
date_str = Calendar.strftime(date, "%Y%m%d")
|
||||
hour_str = hour |> Integer.to_string() |> String.pad_leading(2, "0")
|
||||
fh_str = forecast_hour |> Integer.to_string() |> String.pad_leading(2, "0")
|
||||
|
||||
file =
|
||||
case product do
|
||||
:surface -> "hrrr.t#{hour_str}z.wrfsfcf00.grib2"
|
||||
:pressure -> "hrrr.t#{hour_str}z.wrfprsf00.grib2"
|
||||
:surface -> "hrrr.t#{hour_str}z.wrfsfcf#{fh_str}.grib2"
|
||||
:pressure -> "hrrr.t#{hour_str}z.wrfprsf#{fh_str}.grib2"
|
||||
end
|
||||
|
||||
"#{@hrrr_base}/hrrr.#{date_str}/conus/#{file}"
|
||||
|
|
@ -221,8 +223,8 @@ defmodule Microwaveprop.Weather.HrrrClient do
|
|||
end
|
||||
end
|
||||
|
||||
defp fetch_product_grid(date, hour, product, _points) do
|
||||
url = hrrr_url(date, hour, product)
|
||||
defp fetch_product_grid(date, hour, product, _points, forecast_hour) do
|
||||
url = hrrr_url(date, hour, product, forecast_hour)
|
||||
idx_url = url <> ".idx"
|
||||
|
||||
wanted =
|
||||
|
|
@ -261,9 +263,9 @@ defmodule Microwaveprop.Weather.HrrrClient do
|
|||
":(#{vars}):"
|
||||
end
|
||||
|
||||
defp maybe_fetch_pressure_grid(date, hour, points, opts) do
|
||||
defp maybe_fetch_pressure_grid(date, hour, points, opts, forecast_hour) do
|
||||
if Keyword.get(opts, :pressure, true) do
|
||||
fetch_product_grid(date, hour, :pressure, points)
|
||||
fetch_product_grid(date, hour, :pressure, points, forecast_hour)
|
||||
else
|
||||
{:ok, %{}}
|
||||
end
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
defmodule Microwaveprop.Workers.PropagationGridWorker do
|
||||
@moduledoc """
|
||||
Hourly Oban worker that downloads the latest HRRR data and computes
|
||||
propagation scores across the CONUS grid for all bands.
|
||||
Hourly Oban worker that downloads HRRR data (analysis + forecast hours)
|
||||
and computes propagation scores across the CONUS grid for all bands.
|
||||
Fetches f00-f18 to provide 18-hour forecast timeline.
|
||||
"""
|
||||
|
||||
use Oban.Worker,
|
||||
|
|
@ -18,6 +19,7 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
|
||||
# Pause these queues while the grid fetch runs so they don't compete for bandwidth
|
||||
@pause_queues [:hrrr, :weather, :iemre, :terrain]
|
||||
@max_forecast_hour 18
|
||||
|
||||
@impl Oban.Worker
|
||||
def perform(%Oban.Job{}) do
|
||||
|
|
@ -25,41 +27,70 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
|
||||
# HRRR takes ~45min to publish after the hour. Use 2 hours ago to ensure availability.
|
||||
two_hours_ago = DateTime.add(DateTime.utc_now(), -2, :hour)
|
||||
valid_time = two_hours_ago |> HrrrClient.nearest_hrrr_hour() |> DateTime.truncate(:second)
|
||||
run_time = two_hours_ago |> HrrrClient.nearest_hrrr_hour() |> DateTime.truncate(:second)
|
||||
points = Grid.conus_points()
|
||||
|
||||
Logger.info("PropagationGrid: pausing backfill queues, fetching HRRR for #{valid_time}, #{length(points)} points")
|
||||
Logger.info("PropagationGrid: run_time=#{run_time}, #{length(points)} points, f00-f#{@max_forecast_hour}")
|
||||
|
||||
Enum.each(@pause_queues, &Oban.pause_queue(queue: &1))
|
||||
|
||||
result =
|
||||
with {:ok, grid_data} <- timed("fetch_hrrr", fn -> HrrrClient.fetch_grid(points, valid_time) end) do
|
||||
timed("store_profiles", fn -> store_hrrr_profiles(grid_data, valid_time) end)
|
||||
for_result =
|
||||
for fh <- 0..@max_forecast_hour do
|
||||
valid_time = DateTime.add(run_time, fh * 3600, :second)
|
||||
result = process_forecast_hour(points, run_time, fh, valid_time)
|
||||
|
||||
scores = timed("compute_scores", fn -> compute_scores(grid_data, valid_time) end)
|
||||
|
||||
Logger.info("PropagationGrid: computed #{length(scores)} scores, upserting")
|
||||
|
||||
case timed("upsert_scores", fn -> Propagation.upsert_scores(scores) end) do
|
||||
{:ok, count} ->
|
||||
Logger.info("PropagationGrid: upserted #{count} scores for #{valid_time}")
|
||||
Phoenix.PubSub.broadcast(Microwaveprop.PubSub, "propagation:updated", {:propagation_updated, valid_time})
|
||||
:ok
|
||||
|
||||
error ->
|
||||
Logger.error("PropagationGrid: upsert failed: #{inspect(error)}")
|
||||
error
|
||||
case result do
|
||||
:ok -> valid_time
|
||||
_ -> nil
|
||||
end
|
||||
end
|
||||
|
||||
valid_times = Enum.reject(for_result, &is_nil/1)
|
||||
|
||||
Logger.info("PropagationGrid: resuming backfill queues")
|
||||
Enum.each(@pause_queues, &Oban.resume_queue(queue: &1))
|
||||
|
||||
Weather.prune_old_grid_profiles()
|
||||
Propagation.prune_old_scores()
|
||||
|
||||
if valid_times != [] do
|
||||
Phoenix.PubSub.broadcast(
|
||||
Microwaveprop.PubSub,
|
||||
"propagation:updated",
|
||||
{:propagation_updated, valid_times}
|
||||
)
|
||||
end
|
||||
|
||||
total_ms = System.monotonic_time(:millisecond) - t_start
|
||||
Logger.info("PropagationGrid: total time #{format_duration(total_ms)}")
|
||||
Logger.info("PropagationGrid: total time #{format_duration(total_ms)} (#{length(valid_times)} forecast hours)")
|
||||
|
||||
result
|
||||
:ok
|
||||
end
|
||||
|
||||
defp process_forecast_hour(points, run_time, forecast_hour, valid_time) do
|
||||
label = "f#{String.pad_leading(Integer.to_string(forecast_hour), 2, "0")}"
|
||||
|
||||
case timed(label, fn ->
|
||||
HrrrClient.fetch_grid(points, run_time, forecast_hour: forecast_hour)
|
||||
end) do
|
||||
{:ok, grid_data} ->
|
||||
store_hrrr_profiles(grid_data, valid_time)
|
||||
scores = compute_scores(grid_data, valid_time)
|
||||
|
||||
case Propagation.upsert_scores(scores, prune: false) do
|
||||
{:ok, count} ->
|
||||
Logger.info("PropagationGrid: #{label} → #{count} scores for #{valid_time}")
|
||||
:ok
|
||||
|
||||
error ->
|
||||
Logger.error("PropagationGrid: #{label} upsert failed: #{inspect(error)}")
|
||||
error
|
||||
end
|
||||
|
||||
error ->
|
||||
Logger.warning("PropagationGrid: #{label} fetch failed: #{inspect(error)}")
|
||||
error
|
||||
end
|
||||
end
|
||||
|
||||
defp timed(label, fun) do
|
||||
|
|
@ -114,7 +145,6 @@ defmodule Microwaveprop.Workers.PropagationGridWorker do
|
|||
end
|
||||
end)
|
||||
|
||||
# Batch upsert — skip duplicates
|
||||
stored
|
||||
|> Enum.chunk_every(500)
|
||||
|> Enum.each(fn chunk ->
|
||||
|
|
|
|||
|
|
@ -19,8 +19,10 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
@impl true
|
||||
def mount(_params, _session, socket) do
|
||||
bands = BandConfig.all_bands()
|
||||
valid_time = Propagation.latest_valid_time()
|
||||
initial_scores = Propagation.latest_scores(@default_band, @initial_bounds)
|
||||
valid_times = Propagation.available_valid_times(@default_band)
|
||||
# Default to the earliest (current analysis) hour
|
||||
selected_time = List.first(valid_times)
|
||||
initial_scores = Propagation.scores_at(@default_band, selected_time, @initial_bounds)
|
||||
|
||||
if connected?(socket) do
|
||||
Phoenix.PubSub.subscribe(Microwaveprop.PubSub, "propagation:updated")
|
||||
|
|
@ -32,7 +34,8 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
bands: bands,
|
||||
selected_band: @default_band,
|
||||
initial_scores_json: Jason.encode!(initial_scores),
|
||||
valid_time: valid_time,
|
||||
valid_times: valid_times,
|
||||
selected_time: selected_time,
|
||||
bounds: @initial_bounds,
|
||||
grid_visible: false,
|
||||
antenna_height_ft: 33
|
||||
|
|
@ -42,17 +45,37 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
@impl true
|
||||
def handle_event("select_band", %{"value" => band}, socket) do
|
||||
band = if is_binary(band), do: String.to_integer(band), else: band
|
||||
scores = Propagation.latest_scores(band, socket.assigns.bounds)
|
||||
valid_times = Propagation.available_valid_times(band)
|
||||
selected_time = List.first(valid_times)
|
||||
scores = Propagation.scores_at(band, selected_time, socket.assigns.bounds)
|
||||
|
||||
socket =
|
||||
socket
|
||||
|> assign(:selected_band, band)
|
||||
|> assign(selected_band: band, valid_times: valid_times, selected_time: selected_time)
|
||||
|> push_event("update_scores", %{scores: scores})
|
||||
|> push_event("update_band_info", %{band_info: band_info(band)})
|
||||
|> push_timeline()
|
||||
|
||||
{:noreply, socket}
|
||||
end
|
||||
|
||||
def handle_event("select_time", %{"time" => time_str}, socket) do
|
||||
case DateTime.from_iso8601(time_str) do
|
||||
{:ok, time, _} ->
|
||||
scores = Propagation.scores_at(socket.assigns.selected_band, time, socket.assigns.bounds)
|
||||
|
||||
socket =
|
||||
socket
|
||||
|> assign(:selected_time, time)
|
||||
|> push_event("update_scores", %{scores: scores})
|
||||
|
||||
{:noreply, socket}
|
||||
|
||||
_ ->
|
||||
{:noreply, socket}
|
||||
end
|
||||
end
|
||||
|
||||
def handle_event("set_antenna_height", %{"height_ft" => value}, socket) do
|
||||
height =
|
||||
case Integer.parse(value) do
|
||||
|
|
@ -80,7 +103,7 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
end
|
||||
|
||||
def handle_event("map_bounds", bounds, socket) do
|
||||
scores = Propagation.latest_scores(socket.assigns.selected_band, bounds)
|
||||
scores = Propagation.scores_at(socket.assigns.selected_band, socket.assigns.selected_time, bounds)
|
||||
|
||||
socket =
|
||||
socket
|
||||
|
|
@ -96,7 +119,6 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
freq_ghz = band_mhz / 1_000
|
||||
ant_height_m = socket.assigns.antenna_height_ft * 0.3048
|
||||
|
||||
# Use propagation score to determine atmospheric range potential
|
||||
score =
|
||||
case Propagation.point_detail(band_mhz, lat, lon) do
|
||||
%{score: s} -> s
|
||||
|
|
@ -119,13 +141,24 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
end
|
||||
|
||||
@impl true
|
||||
def handle_info({:propagation_updated, valid_time}, socket) do
|
||||
scores = Propagation.latest_scores(socket.assigns.selected_band, socket.assigns.bounds)
|
||||
def handle_info({:propagation_updated, _valid_times}, socket) do
|
||||
band = socket.assigns.selected_band
|
||||
valid_times = Propagation.available_valid_times(band)
|
||||
# Stay on the same selected time if still available, else pick earliest
|
||||
selected_time =
|
||||
if socket.assigns.selected_time in valid_times do
|
||||
socket.assigns.selected_time
|
||||
else
|
||||
List.first(valid_times)
|
||||
end
|
||||
|
||||
scores = Propagation.scores_at(band, selected_time, socket.assigns.bounds)
|
||||
|
||||
socket =
|
||||
socket
|
||||
|> assign(:valid_time, valid_time)
|
||||
|> assign(valid_times: valid_times, selected_time: selected_time)
|
||||
|> push_event("update_scores", %{scores: scores})
|
||||
|> push_timeline()
|
||||
|
||||
{:noreply, socket}
|
||||
end
|
||||
|
|
@ -149,6 +182,24 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
{:noreply, socket}
|
||||
end
|
||||
|
||||
defp push_timeline(socket) do
|
||||
times =
|
||||
Enum.map(socket.assigns.valid_times, fn t ->
|
||||
%{time: DateTime.to_iso8601(t), label: format_time_label(t)}
|
||||
end)
|
||||
|
||||
selected =
|
||||
if socket.assigns.selected_time,
|
||||
do: DateTime.to_iso8601(socket.assigns.selected_time)
|
||||
|
||||
push_event(socket, "update_timeline", %{times: times, selected: selected})
|
||||
end
|
||||
|
||||
defp format_time_label(dt) do
|
||||
# Show as "HH:MM" in UTC
|
||||
Calendar.strftime(dt, "%H:%M")
|
||||
end
|
||||
|
||||
defp score_range_km(score, config) do
|
||||
cond do
|
||||
score >= 80 -> config.exceptional_range_km
|
||||
|
|
@ -172,17 +223,6 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
}
|
||||
end
|
||||
|
||||
defp time_ago(dt) do
|
||||
seconds = DateTime.diff(DateTime.utc_now(), dt)
|
||||
|
||||
cond do
|
||||
seconds < 60 -> "just now"
|
||||
seconds < 3600 -> "#{div(seconds, 60)}m ago"
|
||||
seconds < 86_400 -> "#{div(seconds, 3600)}h ago"
|
||||
true -> "#{div(seconds, 86_400)}d ago"
|
||||
end
|
||||
end
|
||||
|
||||
defp selected_label(bands, selected_band) do
|
||||
case Enum.find(bands, &(&1.freq_mhz == selected_band)) do
|
||||
nil -> "Select Band"
|
||||
|
|
@ -201,6 +241,14 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
phx-update="ignore"
|
||||
data-scores={@initial_scores_json}
|
||||
data-band-info={Jason.encode!(band_info(@selected_band))}
|
||||
data-valid-times={
|
||||
Jason.encode!(
|
||||
Enum.map(@valid_times, fn t ->
|
||||
%{time: DateTime.to_iso8601(t), label: format_time_label(t)}
|
||||
end)
|
||||
)
|
||||
}
|
||||
data-selected-time={if @selected_time, do: DateTime.to_iso8601(@selected_time), else: ""}
|
||||
class="absolute inset-0 z-0"
|
||||
>
|
||||
</div>
|
||||
|
|
@ -264,11 +312,6 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
<span class="text-xs opacity-70">ft</span>
|
||||
</form>
|
||||
|
||||
<%!-- Latest data --%>
|
||||
<div :if={@valid_time} class="text-xs opacity-70 px-1">
|
||||
Data: {Calendar.strftime(@valid_time, "%b %d, %H:%M UTC")} ({time_ago(@valid_time)})
|
||||
</div>
|
||||
|
||||
<%!-- Links --%>
|
||||
<div class="flex flex-col gap-1 border-t border-base-300 pt-2">
|
||||
<.link navigate="/algo" class="btn btn-xs btn-ghost justify-start gap-1.5">
|
||||
|
|
@ -289,6 +332,14 @@ defmodule MicrowavepropWeb.MapLive do
|
|||
>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<%!-- Bottom timeline bar --%>
|
||||
<div
|
||||
id="forecast-timeline"
|
||||
phx-update="ignore"
|
||||
class="absolute bottom-4 left-1/2 -translate-x-1/2 z-[1000]"
|
||||
>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Layouts.flash_group flash={@flash} />
|
||||
|
|
|
|||
|
|
@ -21,9 +21,7 @@ defmodule Mix.Tasks.PropagationGrid do
|
|||
|
||||
IO.puts("Starting propagation grid computation...")
|
||||
|
||||
case Microwaveprop.Workers.PropagationGridWorker.perform(%Oban.Job{args: %{}}) do
|
||||
:ok -> IO.puts("Done!")
|
||||
{:error, reason} -> IO.puts("Error: #{inspect(reason)}")
|
||||
end
|
||||
Microwaveprop.Workers.PropagationGridWorker.perform(%Oban.Job{args: %{}})
|
||||
IO.puts("Done!")
|
||||
end
|
||||
end
|
||||
|
|
|
|||
202
prediction.md
Normal file
202
prediction.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
# Propagation Prediction System Design
|
||||
|
||||
## The Opportunity
|
||||
|
||||
No tool exists for microwave operators to answer: **"When should I get on the air in the next 24 hours?"**
|
||||
|
||||
Current propagation maps (including ours) show what's happening *right now*. But microwave propagation has a strong, predictable diurnal cycle — ducting peaks near sunrise and collapses in the afternoon. An operator planning a Saturday morning contest attempt needs to know whether 06:00 or 07:30 is the better window, and whether conditions are even worth waking up for.
|
||||
|
||||
We have all the pieces to build this. The scoring algorithm is time-agnostic — feed it conditions for any future hour and it produces a valid score. HRRR provides 18-48 hour forecasts with all the variables we need. And 30% of the composite score (time-of-day + season) is purely deterministic — we can compute it for any moment in the future with perfect accuracy.
|
||||
|
||||
## What We Know About Prediction Accuracy
|
||||
|
||||
### Deterministic Factors (30% of composite weight)
|
||||
|
||||
**Time of Day (20% weight):** The biggest daily swing factor. For 10 GHz:
|
||||
- Sunrise window (±1.5h): score 100
|
||||
- Pre-dawn: 82
|
||||
- Post-sunrise: 78
|
||||
- Evening: 72
|
||||
- Night: 55
|
||||
- Marginal (3-6h after sunrise): 38
|
||||
- Afternoon: 18
|
||||
|
||||
That's an 82-point swing in a single factor worth 20% of the total — driving a ~16-point composite score change from time alone. This is 100% predictable for any future hour.
|
||||
|
||||
**Season (10% weight):** Fixed per month. June scores 90 for 10 GHz; March scores 22. Already known for the entire year.
|
||||
|
||||
### Weather-Dependent Factors (70% of composite weight)
|
||||
|
||||
These require atmospheric observations or forecasts:
|
||||
- **Humidity** (20%): Changes slowly — persists well over 6-12 hours
|
||||
- **Td Depression** (12%): Tied to humidity — similar persistence
|
||||
- **Refractivity** (10%): Most volatile — ducting layers form and dissolve quickly
|
||||
- **Sky Cover** (10%): Moderate persistence; NWP forecasts decent to 12h
|
||||
- **Rain** (8%): Highly variable; HRRR good to ~6h, fair to ~12h
|
||||
- **Wind** (6%): Good persistence to 6h; HRRR skillful to 12h
|
||||
- **Pressure** (4%): Excellent persistence — changes slowly over 12-24h
|
||||
|
||||
### HRRR Forecast Skill by Lead Time
|
||||
|
||||
| Lead Time | Temperature/Dewpoint | Wind | Clouds/Precip | Refractivity |
|
||||
|-----------|---------------------|------|---------------|--------------|
|
||||
| f01-f06 | Excellent | Very good | Good | Fair |
|
||||
| f06-f12 | Good | Good | Fair | Poor |
|
||||
| f12-f18 | Fair | Fair | Marginal | Poor |
|
||||
| f18-f48* | Marginal | Marginal | Poor | Very poor |
|
||||
|
||||
*f18-f48 only available from 00Z and 12Z model runs.
|
||||
|
||||
Refractivity is the hardest factor to forecast. HRRR's 3km grid doesn't fully resolve boundary layer inversions, and ducting events are notoriously chaotic. However, the algorithm is designed to work with imperfect data — the composite score remains informative even with uncertain individual factors.
|
||||
|
||||
## Recommended Architecture: Three Phases
|
||||
|
||||
### Phase 1 — Instant Diurnal Forecast (No new data fetching)
|
||||
|
||||
**What:** When a user clicks a point, immediately compute a 24-hour score timeline using the current weather observation + deterministic time-of-day/season projections.
|
||||
|
||||
**How it works:**
|
||||
1. User clicks point → we already have the current conditions from `point_detail`
|
||||
2. Hold weather factors constant (humidity, Td, refractivity, sky, wind, rain, pressure)
|
||||
3. For each of the next 24 hours, recompute the composite score with:
|
||||
- Weather factors: persisted from current observation
|
||||
- Time of Day: computed for that future hour (deterministic)
|
||||
- Season: same month (deterministic)
|
||||
4. Display as a 24-bar sparkline in the detail panel
|
||||
|
||||
**Why this works:** The diurnal cycle IS the dominant signal. An operator seeing "score rises from 60 to 78 at sunrise" gets 80% of the value without any forecast data. The persistence assumption is reasonable for 6-12 hours under stable weather patterns (which are exactly the conditions that produce good propagation).
|
||||
|
||||
**What operators see:**
|
||||
```
|
||||
24-HOUR OUTLOOK
|
||||
▁▂▃▅▇█▇▅▃▂▁▁▂▃▅▇█▇▅▃▂▁▁
|
||||
Now 6h 12h 18h 24h
|
||||
|
||||
Best window: 06:15-08:30 CDT
|
||||
Peak score: 78/100 (GOOD) — +18 from current
|
||||
Key: Sunrise inversion peak
|
||||
```
|
||||
|
||||
**Implementation cost:** Low. Pure computation — the scorer already handles arbitrary conditions maps. No new data fetching, no schema changes, no new workers. Add a function that loops `composite_score/2` over 24 hours with mutated `utc_hour`/`utc_minute`, render in JS.
|
||||
|
||||
**Accuracy:** Good for identifying WHEN the best window is. Less accurate for absolute score at 12+ hours if weather changes.
|
||||
|
||||
### Phase 2 — HRRR Point Forecast Enhancement
|
||||
|
||||
**What:** When a user clicks a point, also fetch HRRR forecast profiles (f01-f18) for that single lat/lon. Replace persisted weather factors with actual HRRR forecast values.
|
||||
|
||||
**How it works:**
|
||||
1. Modify `hrrr_client.ex` to accept a `forecast_hour` parameter (currently hardcoded to f00)
|
||||
2. On click, `start_async` fetches f01-f18 HRRR profiles for the clicked lat/lon
|
||||
3. Each forecast hour provides: temperature, dewpoint, wind, pressure, clouds, precip, profile levels
|
||||
4. Run `composite_score/2` with actual forecast conditions per hour
|
||||
5. Replace Phase 1's persistence-based timeline with HRRR-enhanced scores
|
||||
|
||||
**What changes for the user:** The timeline becomes more accurate. Instead of "assume current humidity persists," each hour shows what HRRR actually predicts. A cold front moving through at 14Z would show rain scores dropping and pressure rising — Phase 1 wouldn't capture this.
|
||||
|
||||
**Implementation cost:** Medium. Requires:
|
||||
- `hrrr_client.ex`: parameterize forecast_hour in URL builder (~20 lines)
|
||||
- New function to fetch multi-hour forecasts for a single point
|
||||
- LiveView async handler for forecast data
|
||||
- JS timeline update logic
|
||||
- ~2-3 second latency for 18 forecast-hour fetches (byte-range requests in parallel)
|
||||
|
||||
**Accuracy:** Significantly better than Phase 1 for weather-dependent factors, especially f01-f12. Still limited by HRRR's ability to resolve fine-scale refractivity.
|
||||
|
||||
### Phase 3 — Full Grid Forecast Map
|
||||
|
||||
**What:** Pre-compute propagation scores for the entire CONUS grid at key future time steps. The map overlay shows predicted conditions at selectable future hours.
|
||||
|
||||
**How it works:**
|
||||
1. New `ForecastGridWorker` runs after the hourly PropagationGridWorker
|
||||
2. Fetches HRRR f03, f06, f12, f18 (4 key time steps, not all 18)
|
||||
3. Scores all ~20,500 grid points × 8 bands × 4 forecast hours
|
||||
4. Stores in `propagation_scores` with future valid_times
|
||||
5. Map UI adds a time slider/stepper to browse forecast hours
|
||||
6. Animated playback showing propagation evolution
|
||||
|
||||
**What changes for the user:** The entire map overlay shifts to show predicted conditions at future hours. Operators can see "at sunrise tomorrow, where will the best propagation be?" and plan which direction to point their antenna.
|
||||
|
||||
**Implementation cost:** High. Requires:
|
||||
- New Oban worker for forecast grid scoring
|
||||
- Pruning policy update (keep full forecast cycles, not just 2 valid_times)
|
||||
- Schema consideration: add `run_time`/`forecast_hour` columns to distinguish forecasts from analysis
|
||||
- Time slider UI component in the map control panel
|
||||
- Score overlay update mechanism (swap displayed valid_time)
|
||||
- 4× more HRRR data fetching per hour
|
||||
- ~660K additional records per forecast cycle
|
||||
|
||||
**Accuracy:** Same as Phase 2 per-point, but across the full grid. The map-level view is uniquely valuable — no other tool shows predicted microwave propagation conditions geospatially.
|
||||
|
||||
## "Best Time to Operate" — The Killer Feature
|
||||
|
||||
This is the output of Phase 1 that delivers immediate value. For a clicked point:
|
||||
|
||||
**Header:** "Best window: Tomorrow 06:15-08:30 CDT"
|
||||
- Computes the highest-score period from the 24-hour timeline
|
||||
- Shows duration of the window (how long scores stay above threshold)
|
||||
- Shows expected peak score and tier
|
||||
|
||||
**Context:** "Expected score: 78/100 (GOOD) — 18 points above current"
|
||||
- Delta from current score
|
||||
- Primary driver (usually sunrise peak or weather change in Phase 2)
|
||||
|
||||
**Band comparison table:**
|
||||
| Band | Best Window | Peak Score | Current |
|
||||
|------|-----------|------------|---------|
|
||||
| 10 GHz | 06:15-08:30 | 78 (GOOD) | 60 |
|
||||
| 24 GHz | 06:15-07:45 | 71 (GOOD) | 58 |
|
||||
| 47 GHz | 06:30-07:30 | 64 (MARGINAL) | 44 |
|
||||
|
||||
Different bands peak at different times and for different durations — showing this across bands helps operators choose which equipment to set up.
|
||||
|
||||
**Confidence indicator:**
|
||||
- Phase 1: "Based on current conditions + sunrise timing" (honest about persistence assumption)
|
||||
- Phase 2: "Based on HRRR forecast — high confidence to 6h, moderate to 12h"
|
||||
|
||||
## Data Flow Summary
|
||||
|
||||
```
|
||||
Phase 1 (instant):
|
||||
Click → current conditions → loop 24h with deterministic time changes → timeline
|
||||
|
||||
Phase 2 (enhanced):
|
||||
Click → fetch HRRR f01-f18 for point → score each hour → timeline
|
||||
(replaces Phase 1 timeline with actual forecast data)
|
||||
|
||||
Phase 3 (grid):
|
||||
Hourly worker → fetch HRRR f03/f06/f12/f18 for full grid → score all → store
|
||||
Map UI → time slider → load scores for selected valid_time → render overlay
|
||||
```
|
||||
|
||||
## What Makes This Unique
|
||||
|
||||
1. **Microwave-specific:** No general propagation tool models ducting, Fresnel zones, and humidity effects at 10-241 GHz
|
||||
2. **Diurnal-aware:** The sunrise peak is THE event for microwave operators — we quantify exactly when and how strong
|
||||
3. **Band-comparative:** Different bands have inverted seasonal and humidity responses — we show which band to use when
|
||||
4. **Terrain-integrated:** Combined with the viewshed, operators see not just "when" but "where" they can reach
|
||||
5. **Actionable:** "Wake up at 06:15, point your dish northeast, expect GOOD conditions for 2 hours" — that's what operators need
|
||||
|
||||
## Technical Notes
|
||||
|
||||
### Scorer Reuse
|
||||
The `Scorer.composite_score/2` function is completely stateless — it takes a conditions map and a band config, returns a score. For prediction, we just construct conditions maps for future hours and call the same function. No algorithm changes needed.
|
||||
|
||||
### HRRR URL Pattern
|
||||
Current: `hrrr.t{HH}z.wrfsfcf00.grib2`
|
||||
Forecast: `hrrr.t{HH}z.wrfsfcf{FF}.grib2` where FF = 01-48
|
||||
|
||||
The `.idx` index files also exist for forecast hours, so the byte-range fetch strategy works identically.
|
||||
|
||||
### Storage Considerations
|
||||
Phase 3 multiplies storage by ~5× (current 1 valid_time + 4 forecast hours). The covering index on `(band_mhz, valid_time, lat, lon)` already supports efficient queries for any valid_time. Pruning policy needs to shift from "keep 2 most recent valid_times" to "keep the latest complete forecast cycle."
|
||||
|
||||
### Pressure Trend Factor
|
||||
The pressure factor (4% weight) currently compares current vs previous pressure. For predictions:
|
||||
- Phase 1: Use current absolute pressure (trend = nil → absolute scoring)
|
||||
- Phase 2: Can compute trend from consecutive HRRR forecast hours (f05→f06 delta)
|
||||
- Impact is small (4% weight) so this isn't critical.
|
||||
|
||||
## Implementation Priority
|
||||
|
||||
**Phase 1 first.** It delivers 80% of the value with 20% of the effort. The diurnal cycle is the dominant signal, the computation is instant, and the "best time to operate" output is immediately useful. Phase 2 and 3 add accuracy but the core user question — "when should I get on?" — is already answered by Phase 1.
|
||||
Loading…
Add table
Reference in a new issue