Commit graph

3 commits

Author SHA1 Message Date
9bf96349ad
perf(telemetry): cut hot-path overhead on scores_at cache hits
Three small wins from a telemetry audit:

1. Propagation.scores_at/3 wrapped every call in Instrument.span, firing
   two handler dispatches that dominated the ~10µs ETS lookup on cache
   hits. The map's LiveView fires this on every pan + point-click, so
   hot-path latency was mostly telemetry. Span now wraps only the miss
   branch (where disk IO makes the duration signal meaningful); hit path
   still emits the cheap hit/miss counter the cache-ratio panel reads.

2. Oban queue-depth poller: 10s → 30s. Every replica independently
   GROUP-BYs oban_jobs, so each extra replica paid ~3 redundant queries
   per minute for a gauge that moves on the hour-scale anyway.

3. Dropped summary(phoenix.endpoint.start.system_time): summarizing a
   wall-clock timestamp produces no useful aggregate — stop.duration
   below it is the meaningful signal.
2026-04-20 16:12:01 -05:00
2da74c5cd8
feat(telemetry): wide instrumentation + bump hrrr to 2 per pod
Config:
- runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods)

New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with
a [:microwaveprop | event_suffix] prefix so metrics can key off it.

Spans added (each emits duration + result tag):
- HrrrClient.fetch_grid / fetch_profile / fetch_idx
- NexradClient.fetch_frame + decode_png
- IemClient.fetch_asos / fetch_raob
- GefsClient.fetch_grid_profiles
- NarrClient.fetch_profile_at
- UwyoSoundingClient.fetch_sounding
- ElevationClient.fetch_elevation_profile
- Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch
- Propagation.replace_scores
- PropagationGridWorker.compute_scores_algorithm
- TerrainAnalysis.analyse
- CommonVolumeRadarWorker.aggregate_stats

Telemetry catalog (MicrowavepropWeb.Telemetry):
- Oban job duration / queue_time / count / exception by (worker, queue, state)
- Per-span summary metrics for every instrumented phase above
- Periodic (10s) poller emits oban queue depth by (queue, state) — drops
  into the /admin/dashboard Metrics tab immediately

Also drops the now-redundant "fetching n0q frame" and "fetching <url>"
info lines from CommonVolumeRadarWorker / NexradClient; the span events
cover that and the worker's "ingested" line stays for per-job signal.
2026-04-18 16:33:34 -05:00
bc674f0d54
initial 2026-03-28 11:28:47 -05:00