Graham McIntire
2da74c5cd8
Config: - runtime.exs hrrr queue 1 → 2 (6 concurrent HRRR jobs across 3 pods) New helper Microwaveprop.Instrument.span/3 wraps :telemetry.span with a [:microwaveprop | event_suffix] prefix so metrics can key off it. Spans added (each emits duration + result tag): - HrrrClient.fetch_grid / fetch_profile / fetch_idx - NexradClient.fetch_frame + decode_png - IemClient.fetch_asos / fetch_raob - GefsClient.fetch_grid_profiles - NarrClient.fetch_profile_at - UwyoSoundingClient.fetch_sounding - ElevationClient.fetch_elevation_profile - Weather.upsert_hrrr_profiles_batch / upsert_gefs_profiles_batch - Propagation.replace_scores - PropagationGridWorker.compute_scores_algorithm - TerrainAnalysis.analyse - CommonVolumeRadarWorker.aggregate_stats Telemetry catalog (MicrowavepropWeb.Telemetry): - Oban job duration / queue_time / count / exception by (worker, queue, state) - Per-span summary metrics for every instrumented phase above - Periodic (10s) poller emits oban queue depth by (queue, state) — drops into the /admin/dashboard Metrics tab immediately Also drops the now-redundant "fetching n0q frame" and "fetching <url>" info lines from CommonVolumeRadarWorker / NexradClient; the span events cover that and the worker's "ingested" line stays for per-job signal.
26 lines
902 B
Elixir
26 lines
902 B
Elixir
defmodule Microwaveprop.Instrument do
|
|
@moduledoc """
|
|
Thin wrapper around `:telemetry.span/3` so the rest of the codebase can
|
|
instrument a block of work with a single call:
|
|
|
|
Instrument.span([:hrrr, :fetch_grid], %{points: length(points)}, fn ->
|
|
HrrrClient.fetch_grid(points, valid_time)
|
|
end)
|
|
|
|
Emits `[:microwaveprop | event_suffix] ++ [:start|:stop|:exception]`
|
|
with the automatic `duration` measurement, so metrics registered in
|
|
`MicrowavepropWeb.Telemetry` can key off the same prefix.
|
|
"""
|
|
|
|
@type event_suffix :: [atom(), ...]
|
|
|
|
@spec span(event_suffix(), map(), (-> term())) :: term()
|
|
def span(event_suffix, metadata \\ %{}, fun) when is_list(event_suffix) and is_function(fun, 0) do
|
|
event = [:microwaveprop | event_suffix]
|
|
|
|
:telemetry.span(event, metadata, fn ->
|
|
result = fun.()
|
|
{result, Map.put(metadata, :result, :ok)}
|
|
end)
|
|
end
|
|
end
|