The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
- Add one-line moduledocs to 18 LiveViews that were carrying
@moduledoc false. Each line names the route and summarizes what
the page does so a future reader knows where to look before
opening the file.
- Add MapLive handle_event tests for toggle_radar, select_time /
set_selected_time, point_detail, map_bounds, retry_initial_scores.
Assertions focus on socket-state transitions and "didn't crash"
rather than raw HTML — map_bounds in particular has no visible
render-side effect (it push_events the new scores to the JS hook).
- Rust pipeline: integration-shaped test that hand-builds a 2-cell
surface + pressure grid, runs merge → cell_to_conditions →
precompute_band_invariants → composite_score_with → write_atomic
→ decode, asserting header + body length round-trip. Closes the
'pipeline's happy path is only covered by unit tests' gap.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
/live fires every 10s per pod; the request line and Sent 200 response
from Plug.Telemetry drowned out real requests. log_level/1 now returns
false for /live too, same treatment /health already got.
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.
Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
Two-stage mount splits the propagation map's first paint: the static
HTTP render still fetches scores synchronously so SEO/noscript shipping
gets real data baked into `data-scores`, but the websocket-connected
mount defers the potentially-slow fetch to `start_async/3`. Chrome
paints immediately; scores stream in via an `update_scores` push_event
once `handle_async(:initial_scores, ...)` resolves.
A small overlay (spinner on `:loading`, retry button on `:failed`)
uses `<.async_result>` so users can see state and recover from a
transient fetch error.
PromEx.Plugins.Oban counts job exceptions as a Prometheus counter but
does not tell us which worker/queue failed, whether the job is about to
be retried, or the exception class. This adds a telemetry handler on
[:oban, :job, :exception] that:
- emits a single structured Logger.error with worker, queue, job_id,
attempt, max_attempts, retry_exhausted?, kind, reason class, and
args-keys summary (never values — args can contain PII/tokens);
- dispatches opt-in per-worker recovery callbacks
(on_permanent_failure/1 when retries exhausted,
on_transient_failure/1 otherwise);
- catches callback failures so a bug in one worker's recovery path
cannot detach the handler or break logging for other jobs;
- does not emit a second telemetry event, so PromEx counts are not
double-counted.
Handler is attached from Application.start/2 after the Oban supervisor
starts.
Every pod was rescheduling its sweep on a fixed 60 s `Process.send_after`
tick, so N replicas hit the shared NFS `/data/scores` mount and Postgres
in lock-step every minute. Add a uniform jitter of up to 20 s on top of
the 60 s base (effective range 60-80 s) so per-replica schedules drift
apart.
Expose `next_sweep_interval/0,2` as a public helper so the randomized
window can be tested deterministically with `:rand` draws rather than
mocking timers. Existing reschedule test passes `jitter_max_ms: 0` to
keep its 2 s wait window honest.
Previously the `mix deps.patch` task lived at `lib/mix/tasks/deps.patch.ex`
and the `deps.get` alias invoked it. This broke `mix deps.get --only prod`
in Docker: `lib/` isn't compiled yet when deps are fetched, so the task
module couldn't be resolved and the build failed with
`The task "deps.patch" could not be found`.
Move the patch-application function directly into `mix.exs` as a private
helper and wire it into the alias via `&apply_dep_patches/1`. `mix.exs` is
always loadable, so the function is available the moment `deps.get` runs.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
Band-warm failures in the Rust propagation_ready pipeline were only
surfaced via Logger.warning, so Prometheus had no way to see them.
The /weather GridCache hit/miss was likewise invisible while its
/map counterpart (ScoreCache) already reported cache ratio.
- NotifyListener emits [:microwaveprop, :propagation, :notify_listener, :warm]
with %{ok, err} measurements and %{valid_time} metadata after every
propagation_ready notification. Public handle_propagation_ready/1 so
tests can exercise the warm+telemetry path without Postgrex.
- GridCache.fetch/1, fetch_bounds/2, and fetch_point/3 emit
[:microwaveprop, :weather, :grid_cache, :lookup] with a :hit | :miss
metadata key on every ETS lookup.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
IemClient.fetch_asos returns 24-288 rows per station per call, each
previously triggering an individual UPDATE-conflict round-trip via
Weather.upsert_surface_observation/2. On the Turing Pi 2 Postgres
node that per-row latency dominates ingestion time.
Add Weather.upsert_surface_observations/2, a bulk variant that collapses
the rows into a single Repo.insert_all with the same
update-only-when-changed on_conflict predicate and (station_id,
observed_at) conflict target. Switch WeatherFetchWorker to use it.
The remote_ip plug previously trusted Cf-Connecting-Ip and X-Forwarded-For
on every request, so any client reaching a pod directly (cluster-internal,
kubectl port-forward, misconfigured Service) could spoof conn.remote_ip,
which flows into session storage and logs.
Now only honour those headers when the immediate TCP peer sits inside a
configured CIDR list. Default list covers loopback, RFC1918, CGNAT,
link-local, and IPv6 ULA — matches the k8s pod/service network.
Override via config :microwaveprop, :trusted_proxies, or per-environment
via the TRUSTED_PROXY_CIDRS env var in runtime.exs.
Also refresh the stale "nginx/dokku proxy" moduledoc — deployment has
been k8s + Cloudflare for a while.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
Rework of d61fbd3's dialyzer suppression: the 4 warnings from
LiveTable.LiveResource's macro expansion are now fixed at the
root rather than hidden via .dialyzer_ignore.exs.
Changes:
- Delete .dialyzer_ignore.exs and remove ignore_warnings from
mix.exs. No more hidden suppressions.
- New MicrowavepropWeb.LiveTableResource shim that wraps
use LiveTable.LiveResource and overrides:
* maybe_subscribe/1 — explicit :ok = on Phoenix.PubSub.subscribe/2
(previously discarded, tripping :unmatched_return).
* handle_info/2 — explicit _ = on File.rm, Process.send_after.
Switch all 4 LiveTable-using LiveViews (contact_live/index,
beacon_live/index, admin/contact_edit_live, user_management_live/index)
to use MicrowavepropWeb.LiveTableResource instead.
- New priv/dep_patches/live_table-0.4.1.patch adds _ = in front of
LiveTable.LiveSelectHelpers.restore_live_select_from_params/2 in
the dep's macro-expanded handle_params/3. The call's return was
silently discarded, tripping the fourth warning per LiveView.
- New lib/mix/tasks/deps.patch.ex applies every
priv/dep_patches/*.patch to its target dep idempotently after
mix deps.get. Wired into the :setup alias + overridden
mix deps.get so the patch survives cache regeneration in CI.
- Other modified files in this commit are the spec-coverage
additions from the parallel agent pass (beacons.ex,
commercial.ex, propagation.ex, weather.ex, narr_client.ex,
run_timing.ex, 3 workers) — tighter specs for bounds, tuple
shapes, schema-typed params. Dialyzer remains at 0 after
the changes.
Upstream TODO: send a PR for the live_table `_ =` fix. Once
merged and released, drop the patch + mix task + bump the dep.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 2 pre-existing flakes (MapLive timestamp +
PropagationPrune ScoresFile), 0 regressions.
- endpoint.ex log_level/1 now filters by conn.request_path instead of
conn.path_info. Plug.Router.forward/2 (deps/plug/lib/plug.ex:170)
rewrites path_info to the unmatched remainder and extends script_name
with the matched prefix before dispatching to the forwarded plug.
/metrics is routed via forward "/metrics", MetricsPlug; by the time
Plug.Telemetry's before_send callback fires inside MetricsPlug.call/2
the conn it observes has path_info: [] and script_name: ["metrics"],
so the old log_level(%{path_info: ["metrics" | _]}) clause never
matched and the default :info level fired. request_path is set by
the adapter at entry and is never rewritten, making it the correct
discriminator. New regression test in metrics_log_suppression_test.exs
captures both the direct shape and the integration path via Plug.Test.
- Scorer.dbz_to_rain_rate_mmhr/1 now uses a compile-time @mp_inv_b
constant (1 / 1.6) instead of dividing on every rain pixel.
composite_score/2's band-invariant fallback switched from four
separate short-circuits (which silently
mixed cached and freshly-computed values if only some keys were
passed) to a single Map.has_key?/2 branch that honors the "all or
none" contract. Dropped unused band_invariant_tod/1 helper.
- Propagation.replace_scores span scope tightened: the
Instrument.span([:db, :replace_scores]) now wraps only the per-band
ScoresFile.write! loop, not the upstream Enum.group_by grouping
phase. The span name + metadata stay unchanged so Grafana panels
keep working, but the fixed telemetry dispatch cost (~100µs x 2)
is no longer paid for trivially small result sets.
- Added @type t :: %__MODULE__{...} to Accounts.UserToken and
Weather.HrrrClimatology — the last two schemas that lacked one.
Elixir 1.19's set-theoretic inference benefits from every struct
having an explicit t/0 so callers can flow through tightly.
mix dialyzer --format short | grep ^lib/ | wc -l -> 0
mix test: 2165 tests, 3 pre-existing flakes, 0 new regressions.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
1. GridCache: auto-release fill lock when the claimer process crashes.
claim_fill/1 + release_fill/1 go through the GenServer so the
server can Process.monitor the caller and clean up the ETS entry
on :DOWN. Clear/0 now resets both the data table and the lock
table. Fixes a latent bug where a crashed fill leaked the lock
indefinitely, preventing every subsequent /weather mount for that
valid_time from claiming and leaving cache cold.
2. RadarFrameWorker: distinguish permanent vs transient fetch errors.
404 from the IEM n0q archive is permanent (file will never exist)
and marks contacts :unavailable as before. Any other error shape
(5xx, timeout, transport failure) now returns {:error, reason}
so Oban retries — previously those also pinned contacts at
:unavailable after a transient outage.
3. AdminTaskWorker.native_derive: replace per-row Repo.update_all
(N round-trips + N fsyncs) with one UPDATE ... FROM unnest(...)
per 2000-row batch. For the 10k-profile budget this is one
network round trip per chunk instead of 10k, and one fsync per
chunk instead of 10k. Restructured the clause to separate
derivation (pure) from persistence (I/O).
All three changes are test-covered (grid_cache_test auto-release
test, radar_frame_worker_test 5xx + transport tests, existing
admin_task_worker_test native_derive coverage exercises the new
bulk path). Also drops the scorer_diff no-op test that was
verifying the clause removed in 61da51c.
- AdminTaskWorker: remove scorer_diff no-op clause (propagation_scores table
was dropped pre-cutover; queue is drained).
- Weather: remove load_weather_grid_from_db/1 fallback — Rust writes
ProfilesFile for every forecast hour, so the DB path was unreachable
in practice and loaded stale pre-cutover data when it fired.
- Weather.build_grid_cache_row/4 now prefers persisted fields from the
ProfilesFile profile map (surface_refractivity, min_refractivity_gradient,
ducting_detected, duct_characteristics) over re-deriving from the raw
profile, matching the old DB-path semantics.
- NotifyListener: wrap per-band warm in try/rescue so one corrupt or
mid-rename .ntms file doesn't abort the remaining bands. Failures log
a warning with band + valid_time; successes still emit one info line.
- weather_grid_test: rewrite insert_hrrr_profile helper to write
ProfilesFile directly and drop the "only returns points on 0.125 grid"
test (filter was a property of the deleted DB fallback; Rust writes
only grid points by construction).
Column name now matches the type key used everywhere else:
:hrrr -> hrrr_status, :weather -> weather_status,
:terrain -> terrain_status, :mechanism -> mechanism_status
Drops the @status_column_overrides remap in BackfillEnqueueWorker
(status_column/1 is now a straight :"#{type}_status" derivation) and
the long-form references in Contact, ContactWeatherEnqueueWorker,
and MechanismClassifyWorker. Migration is a metadata-only RENAME
COLUMN + RENAME INDEX — safe to land in prod (no table rewrite,
brief catalog lock only).
Periodic 60s sweep re-warms ScoreCache from /data/scores whenever
{band, valid_time} pairs are on disk but absent in local ETS.
Covers the three silent-fail modes of the NOTIFY/LISTEN path:
dropped notifies on reconnect, missing listener, and stale NFS
reads. Each pod reconciles its own cache — no PubSub fan-out,
no coordination required. Rescue on File.Error handles the
rare window where an atomic-rename write races the reader.
docs/runbook_propagation_pipeline.md names every failure mode
in the Rust↔Elixir seam and what recovers it.
Pure domain check — "does this contact already have HRRR profiles at
every path point for its QSO hour" — has no business living in an
Oban worker. Weather already owns has_hrrr_profile?/3 and
round_to_hrrr_grid/2, so consolidation keeps HRRR knowledge in one
context and leaves ContactWeatherEnqueueWorker focused on job
coordination.
Phase 2 cutover completed in 65693ed — Rust prop-grid-rs owns
f01-f18 production writes and the shadow path has been dead
for weeks. No callers in lib/, test/, or mix tasks.
Every Sent log line now carries method and path so noisy traffic
(probes, scrapers, redirects) is identifiable without a tracer.
RemoteIp plug already sets remote_ip metadata; add method and
request_path alongside it and register the extra keys with the
default formatter.
hrrr_placeholder_jobs previously returned [contact.id] for any
post-2014 contact with pos1, regardless of whether the HRRR
profiles were already in the DB. That meant BackfillEnqueueWorker
re-flagged the same ~22k contacts as :queued on every 30 min
cron tick forever — HrrrPointEnqueuer saw 0 missing points and
inserted nothing, but the status field never advanced.
Now the placeholder also returns [] when every path point has
an hrrr_profile row, so mark_hrrr_status!/3's empty-list branch
flips the contact to :complete directly. Genuinely-missing
contacts still flow through HrrrPointEnqueuer unchanged.
count_unprocessed_raw fired seven separate COUNT(*) scans on the
contacts table every 2 s refresh (total, terrain, hrrr, weather, iemre,
radar, all-done). Each one went through the 58k+ row table
independently, so the status page was hitting Postgres with ~3.5 full
scans per second of connected viewer time.
A single query with COUNT(*) FILTER clauses pulls every count in one
sequential scan. Same result, roughly 1/7 the CPU on Postgres.
Two wins in the /map hot path.
1. preload_forecast fired on every Leaflet moveend — which arrives in
bursts during pan/zoom — and each firing read + filtered + shipped
18 forecast-hour score lists (up to 90k cells per hour at full
CONUS view) over the LiveView websocket. Now debounced to the
trailing edge: the user must stop moving for 750 ms before we pay
the preload cost. select_band and propagation_updated go through
the same scheduler so a storm of PubSub updates during an hourly
chain also coalesces.
2. ScoreCache.grid_to_filtered_list walked the 92k-cell grid twice
(Enum.filter then Enum.map) and allocated an intermediate tuple
list between them. Enum.reduce emits only in-bounds result maps
directly — halves map traversal + drops the tuple intermediate.
The Contact list's 'Partial' badge was sticky: once a worker gave up on
a contact (upstream 404, no nearby station, IEM throttling) or the
stale-queued reconciler flipped it to :unavailable after 3 days, the
backfill cron never looked at it again. Upstream archives heal over
time — IEM backfills gaps, new ASOS stations come online, Rust worker
gains capability — so rows stayed 'Partial' that shouldn't have.
Changes:
* BackfillEnqueueWorker.type_filter now also matches <status>_status =
:unavailable when contacts.updated_at is older than a 24 h cooldown.
enqueue_for_contact is idempotent: if there's still nothing to
fetch it'll flip the status to :complete on the empty-jobs branch;
if new data shows up it builds fresh jobs.
* reconcile_stale_queued_to_unavailable now bumps updated_at when
flipping to :unavailable so the cooldown starts fresh — otherwise
the main loop would pick it right back up on the same cron tick
(stale updated_at < any reasonable cooldown) and immediately
transition :unavailable → :complete, defeating the :unavailable
signal entirely.
Also:
* Drop ScoresFile.point_score/3, dead since read_point moved to the
pread fast path in commit bd3b114.
* Clear Microwaveprop.Cache in WeatherGridTest setup. The new
ProfilesFile read cache is keyed by (base_dir, valid_time); tests
in this file reuse DateTime.utc_now() truncated to seconds and
share the default base_dir, so a prior test's cached read leaks
into the next one's expectations.
The 'Fetching HRRR atmospheric data' spinner stayed up forever for
contacts stuck in :queued without a matching hrrr_fetch_tasks row.
Ways a contact lands in that state:
- HrrrPointEnqueuer.enqueue/1 raises, the rescue swallows the error
and returns {:ok, 0}, but the caller still marked the contact
:queued before it noticed the failure.
- Rust worker marks a (valid_time) task :done with a points array
that never included this contact's point, so no profile is ever
written for it.
- hrrr_fetch_tasks row is cleaned up by retention before the contact
page loads.
Drop the hrrr_status != :queued short-circuit and always call the
enqueuer when the profile is nil. The ON CONFLICT clause in
HrrrPointEnqueuer.enqueue/1 unions points into the existing row,
resets :done/:failed rows back to :queued with attempt=0, and no-ops
when the point is already present — so re-entering on every page view
is cheap and self-healing.
Clicks on the map were hanging for 1-3s, triggering the Phoenix topbar
progress bar visible to the user. Three hot-path improvements:
1. ScoresFile.read_point now uses a two-range :file.pread/2 — reads the
33-byte header + one cell byte (~60 bytes) instead of the full ~93 KB
file. point_forecast walks 19 forecast hours per click, so the old
full-read path was the dominant cost on cold cache.
2. ProfilesFile.read/1 and list_valid_times/0 cached via
Microwaveprop.Cache (5s TTL). Decoded profile maps are ~10 MB each
(92k cells); timeline scrub that re-clicks the same valid_time
within seconds now hits ETS instead of re-gunzipping + decoding.
Cache keys include base_dir so test setup that swaps dirs doesn't
see stale entries. Writes + prune + retain_window invalidate.
3. Endpoint log_level now filters /metrics the same way it already
filters /health — Prometheus scrapes every 5s and was producing
visible log spam.
Pod-local ETS is right for this workload (per-click read, tiny working
set); Valkey / shared memstore would not help here since each pod needs
its own fast-path lookup. File a follow-up if cross-pod score lookups
ever show up in flame graphs.
Three small wins from a telemetry audit:
1. Propagation.scores_at/3 wrapped every call in Instrument.span, firing
two handler dispatches that dominated the ~10µs ETS lookup on cache
hits. The map's LiveView fires this on every pan + point-click, so
hot-path latency was mostly telemetry. Span now wraps only the miss
branch (where disk IO makes the duration signal meaningful); hit path
still emits the cheap hit/miss counter the cache-ratio panel reads.
2. Oban queue-depth poller: 10s → 30s. Every replica independently
GROUP-BYs oban_jobs, so each extra replica paid ~3 redundant queries
per minute for a gauge that moves on the hour-scale anyway.
3. Dropped summary(phoenix.endpoint.start.system_time): summarizing a
wall-clock timestamp produces no useful aggregate — stop.duration
below it is the meaningful signal.
Contacts lacking pos2 had terrain/radar/mechanism stuck on :pending, so
BackfillEnqueueWorker re-scanned them every cron cycle. Mark these
statuses :unavailable so the enqueue filter excludes them; a later pos2
edit resets them back to :pending via Radio.reset_enrichment_statuses/2.
BackfillEnqueueWorker re-enqueued every :queued contact each cycle, and
RadarFrameWorker had no unique clause, so the same 5-min NEXRAD frame
got scheduled repeatedly with overlapping contact lists. Prod had 9,134
duplicate available jobs across 1,061 distinct frames — 89% waste.
Two coupled changes so contacts aren't stranded by the dedup:
1. Worker now queries eligible contacts by the frame's 5-min bucket
instead of the args' contact_ids list. A contact that flips to
:queued between the enqueue and the job running gets picked up.
2. unique: [frame_ts] on states [available, scheduled, retryable].
:executing intentionally excluded — a contact that flips to :queued
mid-execution can still enqueue a follow-up instead of getting
stuck behind a running job that missed it.
HRRR publishes ~2h after the cycle hour, so the hourly seeder uses
run_time = now - 2h. That puts the f00 analysis file's valid_time
right at the 2h prune cutoff — the file was being deleted within
minutes of the Rust analysis step writing it, leaving /weather and
/map point-detail with no data.
A 3h cutoff keeps the current analysis alive until the next hourly
run supersedes it; forecast files (valid_time >> cutoff) are
unaffected.
Second row in the Rust worker table pulls from hrrr_fetch_tasks so
per-QSO HRRR backfill progress is visible alongside prop-grid-rs.
Same columns (Running / Queued / Retrying / Done 1h), 2s cache, same
refresh debounce as the grid_tasks panel.
