Band-warm failures in the Rust propagation_ready pipeline were only
surfaced via Logger.warning, so Prometheus had no way to see them.
The /weather GridCache hit/miss was likewise invisible while its
/map counterpart (ScoreCache) already reported cache ratio.
- NotifyListener emits [:microwaveprop, :propagation, :notify_listener, :warm]
with %{ok, err} measurements and %{valid_time} metadata after every
propagation_ready notification. Public handle_propagation_ready/1 so
tests can exercise the warm+telemetry path without Postgrex.
- GridCache.fetch/1, fetch_bounds/2, and fetch_point/3 emit
[:microwaveprop, :weather, :grid_cache, :lookup] with a :hit | :miss
metadata key on every ETS lookup.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
Set OTEL_EXPORTER_OTLP_ENDPOINT on the prop deployment to the cluster
Collector so Phoenix/Bandit/Oban spans export and link up with
downstream prop-grid-rs / prop-hrrr-point-rs traces in Tempo. Without
this the Elixir app fell back to traces_exporter :none and the Rust
worker spans had no parent, leaving the trace graph disconnected.
The stale comment referenced opentelemetry_ecto as the reason export
was off, but that package was never added to mix.exs (only
opentelemetry_phoenix / _bandit / _oban are listed), so the
span-flood concern no longer applies.
Weather.find_nearest_hrrr/3 and Weather.find_nearest_native_profile/3
run a three-range scan gated first by a ±1h valid_time window, then
by a ±0.07° lat/lon box. The existing unique `(lat, lon, valid_time)`
index is only usable on its `lat` prefix for these queries, forcing
Postgres to scan wide lat slices before applying the valid_time
filter — expensive on the Turing Pi 2 node.
Add a `(valid_time, lat, lon)` composite so valid_time leads,
collapsing the search to a narrow btree slice inside a single
partition before the lat/lon box filter kicks in.
`hrrr_profiles` is RANGE-partitioned by valid_time, so
CREATE INDEX CONCURRENTLY is applied per partition and the resulting
indexes are attached to a parent index created ON ONLY the partitioned
table. `hrrr_native_profiles` is a regular table, so the concurrent
build runs directly on it.
IemClient.fetch_asos returns 24-288 rows per station per call, each
previously triggering an individual UPDATE-conflict round-trip via
Weather.upsert_surface_observation/2. On the Turing Pi 2 Postgres
node that per-row latency dominates ingestion time.
Add Weather.upsert_surface_observations/2, a bulk variant that collapses
the rows into a single Repo.insert_all with the same
update-only-when-changed on_conflict predicate and (station_id,
observed_at) conflict target. Switch WeatherFetchWorker to use it.
The remote_ip plug previously trusted Cf-Connecting-Ip and X-Forwarded-For
on every request, so any client reaching a pod directly (cluster-internal,
kubectl port-forward, misconfigured Service) could spoof conn.remote_ip,
which flows into session storage and logs.
Now only honour those headers when the immediate TCP peer sits inside a
configured CIDR list. Default list covers loopback, RFC1918, CGNAT,
link-local, and IPv6 ULA — matches the k8s pod/service network.
Override via config :microwaveprop, :trusted_proxies, or per-environment
via the TRUSTED_PROXY_CIDRS env var in runtime.exs.
Also refresh the stale "nginx/dokku proxy" moduledoc — deployment has
been k8s + Cloudflare for a while.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
- MapLiveTest timestamp indicator: earlier tests in the suite leave
score files in the shared `:propagation_scores_dir` and warm the 5 s
`list_valid_times` cache; clear both in setup so the "No data"
assertion sees the state the test documents.
- PropagationPruneWorkerTest: `old = now - 3h` collides with the
worker's cutoff (also `now - 3h`) once both are truncated to the
second, so they land equal when test + worker run in the same wall
clock second. Shift `old` by an extra -1s for unconditional ordering.
- ContactWeatherEnqueueWorkerTest: flip to async: false — perform/1
runs inline Oban child jobs that upsert rows with a shared conflict
target, and parallel DataCase siblings hit "ShareLock deadlock_detected"
intermittently.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
- endpoint.ex log_level/1 now filters by conn.request_path instead of
conn.path_info. Plug.Router.forward/2 (deps/plug/lib/plug.ex:170)
rewrites path_info to the unmatched remainder and extends script_name
with the matched prefix before dispatching to the forwarded plug.
/metrics is routed via forward "/metrics", MetricsPlug; by the time
Plug.Telemetry's before_send callback fires inside MetricsPlug.call/2
the conn it observes has path_info: [] and script_name: ["metrics"],
so the old log_level(%{path_info: ["metrics" | _]}) clause never
matched and the default :info level fired. request_path is set by
the adapter at entry and is never rewritten, making it the correct
discriminator. New regression test in metrics_log_suppression_test.exs
captures both the direct shape and the integration path via Plug.Test.
- Scorer.dbz_to_rain_rate_mmhr/1 now uses a compile-time @mp_inv_b
constant (1 / 1.6) instead of dividing on every rain pixel.
composite_score/2's band-invariant fallback switched from four
separate short-circuits (which silently
mixed cached and freshly-computed values if only some keys were
passed) to a single Map.has_key?/2 branch that honors the "all or
none" contract. Dropped unused band_invariant_tod/1 helper.
- Propagation.replace_scores span scope tightened: the
Instrument.span([:db, :replace_scores]) now wraps only the per-band
ScoresFile.write! loop, not the upstream Enum.group_by grouping
phase. The span name + metadata stay unchanged so Grafana panels
keep working, but the fixed telemetry dispatch cost (~100µs x 2)
is no longer paid for trivially small result sets.
- Added @type t :: %__MODULE__{...} to Accounts.UserToken and
Weather.HrrrClimatology — the last two schemas that lacked one.
Elixir 1.19's set-theoretic inference benefits from every struct
having an explicit t/0 so callers can flow through tightly.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 3 pre-existing flakes, 0 new regressions.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
1. GridCache: auto-release fill lock when the claimer process crashes.
claim_fill/1 + release_fill/1 go through the GenServer so the
server can Process.monitor the caller and clean up the ETS entry
on :DOWN. Clear/0 now resets both the data table and the lock
table. Fixes a latent bug where a crashed fill leaked the lock
indefinitely, preventing every subsequent /weather mount for that
valid_time from claiming and leaving cache cold.
2. RadarFrameWorker: distinguish permanent vs transient fetch errors.
404 from the IEM n0q archive is permanent (file will never exist)
and marks contacts :unavailable as before. Any other error shape
(5xx, timeout, transport failure) now returns {:error, reason}
so Oban retries — previously those also pinned contacts at
:unavailable after a transient outage.
3. AdminTaskWorker.native_derive: replace per-row Repo.update_all
(N round-trips + N fsyncs) with one UPDATE ... FROM unnest(...)
per 2000-row batch. For the 10k-profile budget this is one
network round trip per chunk instead of 10k, and one fsync per
chunk instead of 10k. Restructured the clause to separate
derivation (pure) from persistence (I/O).
All three changes are test-covered (grid_cache_test auto-release
test, radar_frame_worker_test 5xx + transport tests, existing
admin_task_worker_test native_derive coverage exercises the new
bulk path). Also drops the scorer_diff no-op test that was
verifying the clause removed in 61da51c.
- AdminTaskWorker: remove scorer_diff no-op clause (propagation_scores table
was dropped pre-cutover; queue is drained).
- Weather: remove load_weather_grid_from_db/1 fallback — Rust writes
ProfilesFile for every forecast hour, so the DB path was unreachable
in practice and loaded stale pre-cutover data when it fired.
- Weather.build_grid_cache_row/4 now prefers persisted fields from the
ProfilesFile profile map (surface_refractivity, min_refractivity_gradient,
ducting_detected, duct_characteristics) over re-deriving from the raw
profile, matching the old DB-path semantics.
- NotifyListener: wrap per-band warm in try/rescue so one corrupt or
mid-rename .ntms file doesn't abort the remaining bands. Failures log
a warning with band + valid_time; successes still emit one info line.
- weather_grid_test: rewrite insert_hrrr_profile helper to write
ProfilesFile directly and drop the "only returns points on 0.125 grid"
test (filter was a property of the deleted DB fallback; Rust writes
only grid points by construction).
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
Periodic 60s sweep re-warms ScoreCache from /data/scores whenever
{band, valid_time} pairs are on disk but absent in local ETS.
Covers the three silent-fail modes of the NOTIFY/LISTEN path:
dropped notifies on reconnect, missing listener, and stale NFS
reads. Each pod reconciles its own cache — no PubSub fan-out,
no coordination required. Rescue on File.Error handles the
rare window where an atomic-rename write races the reader.
docs/runbook_propagation_pipeline.md names every failure mode
in the Rust↔Elixir seam and what recovers it.
Pure domain check — "does this contact already have HRRR profiles at
every path point for its QSO hour" — has no business living in an
Oban worker. Weather already owns has_hrrr_profile?/3 and
round_to_hrrr_grid/2, so consolidation keeps HRRR knowledge in one
context and leaves ContactWeatherEnqueueWorker focused on job
coordination.
Phase 2 cutover completed in 65693ed — Rust prop-grid-rs owns
f01-f18 production writes and the shadow path has been dead
for weeks. No callers in lib/, test/, or mix tasks.
Every Sent log line now carries method and path so noisy traffic
(probes, scrapers, redirects) is identifiable without a tracer.
RemoteIp plug already sets remote_ip metadata; add method and
request_path alongside it and register the extra keys with the
default formatter.
hrrr_placeholder_jobs previously returned [contact.id] for any
post-2014 contact with pos1, regardless of whether the HRRR
profiles were already in the DB. That meant BackfillEnqueueWorker
re-flagged the same ~22k contacts as :queued on every 30 min
cron tick forever — HrrrPointEnqueuer saw 0 missing points and
inserted nothing, but the status field never advanced.
Now the placeholder also returns [] when every path point has
an hrrr_profile row, so mark_hrrr_status!/3's empty-list branch
flips the contact to :complete directly. Genuinely-missing
contacts still flow through HrrrPointEnqueuer unchanged.
Unlimited BackfillEnqueueWorker runs loaded all ~36k enrichable
contacts and hit Oban's ~17 min execution timeout, discarding the
parent job every cron tick. Enrichment was landing partially
(jobs enqueued before the timeout) but the loop never completed
and no success signal was emitted. A per-tick limit of 2000 keeps
each run well under any timeout while still draining the backlog
at 4000/hour across the */30 schedule.
count_unprocessed_raw fired seven separate COUNT(*) scans on the
contacts table every 2 s refresh (total, terrain, hrrr, weather, iemre,
radar, all-done). Each one went through the 58k+ row table
independently, so the status page was hitting Postgres with ~3.5 full
scans per second of connected viewer time.
A single query with COUNT(*) FILTER clauses pulls every count in one
sequential scan. Same result, roughly 1/7 the CPU on Postgres.
Two wins in the /map hot path.
1. preload_forecast fired on every Leaflet moveend — which arrives in
bursts during pan/zoom — and each firing read + filtered + shipped
18 forecast-hour score lists (up to 90k cells per hour at full
CONUS view) over the LiveView websocket. Now debounced to the
trailing edge: the user must stop moving for 750 ms before we pay
the preload cost. select_band and propagation_updated go through
the same scheduler so a storm of PubSub updates during an hourly
chain also coalesces.
2. ScoreCache.grid_to_filtered_list walked the 92k-cell grid twice
(Enum.filter then Enum.map) and allocated an intermediate tuple
list between them. Enum.reduce emits only in-bounds result maps
directly — halves map traversal + drops the tuple intermediate.
The Contact list's 'Partial' badge was sticky: once a worker gave up on
a contact (upstream 404, no nearby station, IEM throttling) or the
stale-queued reconciler flipped it to :unavailable after 3 days, the
backfill cron never looked at it again. Upstream archives heal over
time — IEM backfills gaps, new ASOS stations come online, Rust worker
gains capability — so rows stayed 'Partial' that shouldn't have.
Changes:
* BackfillEnqueueWorker.type_filter now also matches <status>_status =
:unavailable when contacts.updated_at is older than a 24 h cooldown.
enqueue_for_contact is idempotent: if there's still nothing to
fetch it'll flip the status to :complete on the empty-jobs branch;
if new data shows up it builds fresh jobs.
* reconcile_stale_queued_to_unavailable now bumps updated_at when
flipping to :unavailable so the cooldown starts fresh — otherwise
the main loop would pick it right back up on the same cron tick
(stale updated_at < any reasonable cooldown) and immediately
transition :unavailable → :complete, defeating the :unavailable
signal entirely.
Also:
* Drop ScoresFile.point_score/3, dead since read_point moved to the
pread fast path in commit bd3b114.
* Clear Microwaveprop.Cache in WeatherGridTest setup. The new
ProfilesFile read cache is keyed by (base_dir, valid_time); tests
in this file reuse DateTime.utc_now() truncated to seconds and
share the default base_dir, so a prior test's cached read leaks
into the next one's expectations.
The 'Fetching HRRR atmospheric data' spinner stayed up forever for
contacts stuck in :queued without a matching hrrr_fetch_tasks row.
Ways a contact lands in that state:
- HrrrPointEnqueuer.enqueue/1 raises, the rescue swallows the error
and returns {:ok, 0}, but the caller still marked the contact
:queued before it noticed the failure.
- Rust worker marks a (valid_time) task :done with a points array
that never included this contact's point, so no profile is ever
written for it.
- hrrr_fetch_tasks row is cleaned up by retention before the contact
page loads.
Drop the hrrr_status != :queued short-circuit and always call the
enqueuer when the profile is nil. The ON CONFLICT clause in
HrrrPointEnqueuer.enqueue/1 unions points into the existing row,
resets :done/:failed rows back to :queued with attempt=0, and no-ops
when the point is already present — so re-entering on every page view
is cheap and self-healing.
Clicks on the map were hanging for 1-3s, triggering the Phoenix topbar
progress bar visible to the user. Three hot-path improvements:
1. ScoresFile.read_point now uses a two-range :file.pread/2 — reads the
33-byte header + one cell byte (~60 bytes) instead of the full ~93 KB
file. point_forecast walks 19 forecast hours per click, so the old
full-read path was the dominant cost on cold cache.
2. ProfilesFile.read/1 and list_valid_times/0 cached via
Microwaveprop.Cache (5s TTL). Decoded profile maps are ~10 MB each
(92k cells); timeline scrub that re-clicks the same valid_time
within seconds now hits ETS instead of re-gunzipping + decoding.
Cache keys include base_dir so test setup that swaps dirs doesn't
see stale entries. Writes + prune + retain_window invalidate.
3. Endpoint log_level now filters /metrics the same way it already
filters /health — Prometheus scrapes every 5s and was producing
visible log spam.
Pod-local ETS is right for this workload (per-click read, tiny working
set); Valkey / shared memstore would not help here since each pod needs
its own fast-path lookup. File a follow-up if cross-pod score lookups
ever show up in flame graphs.
Three small wins from a telemetry audit:
1. Propagation.scores_at/3 wrapped every call in Instrument.span, firing
two handler dispatches that dominated the ~10µs ETS lookup on cache
hits. The map's LiveView fires this on every pan + point-click, so
hot-path latency was mostly telemetry. Span now wraps only the miss
branch (where disk IO makes the duration signal meaningful); hit path
still emits the cheap hit/miss counter the cache-ratio panel reads.
2. Oban queue-depth poller: 10s → 30s. Every replica independently
GROUP-BYs oban_jobs, so each extra replica paid ~3 redundant queries
per minute for a gauge that moves on the hour-scale anyway.
3. Dropped summary(phoenix.endpoint.start.system_time): summarizing a
wall-clock timestamp produces no useful aggregate — stop.duration
below it is the meaningful signal.
Contacts lacking pos2 had terrain/radar/mechanism stuck on :pending, so
BackfillEnqueueWorker re-scanned them every cron cycle. Mark these
statuses :unavailable so the enqueue filter excludes them; a later pos2
edit resets them back to :pending via Radio.reset_enrichment_statuses/2.
