Commit graph

6 commits

Author SHA1 Message Date
f26fbafc29
fix: log every silent error path so failures surface in k8s logs
Sweep of remaining swallowed-error sites flagged by an audit pass:

- weather/gefs_client.ex: byte-range download task exits log url + reason
  before being surfaced as {:error, _} (matches the HRRR fix).
- live/path_live.ex: terrain-profile load failure, native-duct lookup
  failure, and ProfilesFile.read failure now log warnings with path
  endpoints / midpoint / valid_time. Previously rendered a degraded
  result silently.
- live/rover_live.ex: station-mutation {:error, _} now logs id + reason
  instead of dropping into {:noreply, socket}.
- application.ex: the boot-time backfill_missing_home_qth Task.start is
  now wrapped in try/rescue. A DB issue at boot would have crashed the
  task silently (no Logger handler attached pre-Logger crash format).
2026-04-27 14:31:30 -05:00
15ce50365f
perf(concurrency): partition Task.Supervisor to remove bottleneck on heavy async_stream paths
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.

Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
2026-04-21 14:20:29 -05:00
2da74c5cd8
feat(telemetry): wide instrumentation + bump hrrr to 2 per pod
Config:
- runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods)

New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with
a [:microwaveprop | event_suffix] prefix so metrics can key off it.

Spans added (each emits duration + result tag):
- HrrrClient.fetch_grid / fetch_profile / fetch_idx
- NexradClient.fetch_frame + decode_png
- IemClient.fetch_asos / fetch_raob
- GefsClient.fetch_grid_profiles
- NarrClient.fetch_profile_at
- UwyoSoundingClient.fetch_sounding
- ElevationClient.fetch_elevation_profile
- Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch
- Propagation.replace_scores
- PropagationGridWorker.compute_scores_algorithm
- TerrainAnalysis.analyse
- CommonVolumeRadarWorker.aggregate_stats

Telemetry catalog (MicrowavepropWeb.Telemetry):
- Oban job duration / queue_time / count / exception by (worker, queue, state)
- Per-span summary metrics for every instrumented phase above
- Periodic (10s) poller emits oban queue depth by (queue, state) — drops
  into the /admin/dashboard Metrics tab immediately

Also drops the now-redundant "fetching n0q frame" and "fetching <url>"
info lines from CommonVolumeRadarWorker / NexradClient; the span events
cover that and the worker's "ingested" line stays for per-job signal.
2026-04-18 16:33:34 -05:00
4d0c15e3b8
feat(gefs): score extended-horizon grid and seed cron
Wires the GEFS fetch pipeline into the existing scoring machinery:

- GefsFetchWorker runs Propagation.score_grid_point over every
  fetched grid cell and persists the result via replace_scores/2, so
  Day 2-7 valid_times show up on the map through the same
  Propagation.scores_at/3 path HRRR uses
- Empty-args perform/1 seeds the chain by enqueueing f024-f168 (25
  jobs at 6-hour cadence) for the most recent GEFS run that should
  be published given NOMADS' ~3-4h publication lag
- Cron: seeds at 05:30, 11:30, 17:30, 23:30 UTC — 5 hours after each
  00/06/12/18Z run
- GefsClient.build_profile now emits :wind_u / :wind_v atom keys to
  match the scorer's input shape; DB columns keep the *_mps suffix
  for unit clarity, remapped at write time

GEFS pgrb2a lacks HPBL, native-level duct data, NEXRAD, and
commercial-link degradation — the extended-horizon score is a
rougher signal than the HRRR-driven one but covers the window HRRR
can't reach.
2026-04-18 14:41:41 -05:00
0537a1831b
feat(gefs): ingest worker and grid-profile extraction
- GefsClient.build_profile/1: converts raw wgrib2 output into the
  scorer-shaped profile, deriving Td from RH via Magnus at both the
  surface and each pressure level
- GefsClient.fetch_grid_profiles/3: downloads the idx sidecar from
  NOMADS, byte-ranges only the surface messages, hands the slim
  GRIB2 binary to wgrib2 -lola for bilinear regridding from 0.5°
  onto the CONUS 0.125° grid
- GefsFetchWorker: one Oban job per (run_time, forecast_hour),
  upserts into gefs_profiles and runs SoundingParams.derive so the
  same refractivity/ducting metrics flow through

Oban gets a new :gefs queue (2 slots dev/config, 1 slot per prod
pod). Test env gets a gefs_req_options Req.Test stub slot so future
integration tests can drive the full fetch path. No cron wiring or
scorer integration yet — that's Step 3.
2026-04-18 14:36:03 -05:00
a5c7dce147
feat(gefs): scaffold extended-horizon forecast ingestion
Lays the groundwork for a Day 2-7 propagation outlook driven by NCEP
GEFS ensemble-mean output, picking up where HRRR's 18-hour horizon
leaves off.

- GefsClient: NOMADS URL builder, forecast-hour list (3h to +240,
  6h to +384), ensemble-mean message inventory, and a Magnus
  dewpoint derivation (pgrb2a publishes RH rather than Td)
- gefs_profiles table + GefsProfile schema mirroring hrrr_profiles
  so the scorer can run over rows from either source
- Weather.upsert_gefs_profile/1 and upsert_gefs_profiles_batch/1

No worker or UI yet — those land in follow-up commits.
2026-04-18 14:28:44 -05:00