Two wins that directly shrink /map and /weather latency under
interactive use:
**Debounce the moveend → repaint path (150ms).** Leaflet fires
`moveend` once per pan but 5–10× per second during a wheel-zoom.
Previously each one ran a full `scores_at` (or `weather_grid_at`)
read + a ~20–100KB websocket push. Now they coalesce into a single
trailing-edge flush via `schedule_bounds_update` (and the
/weather twin `schedule_weather_flush`) — the same pattern we
already use for forecast preload. 150ms is short enough that the
user doesn't perceive the pause; >90% of the burst disappears.
**Parallelize point_forecast across 4 tasks.** The sparkline that
pops up on point click was reading up to 18 `.prop` files
sequentially off NFS. Each read is cheap (pread one byte at
row*cols+col) but RTT-dominated. `Task.async_stream` with
max_concurrency=4, ordered=true brings 18-file wall time from
~50ms down to ~15ms on NFS without changing the public API.
No test changes: existing map_bounds test only asserts the bounds
assign lands, which the debounced path still does synchronously.
The main /map timeline was rendering each forecast hour twice
('Now 22:00' + '0h 22:00', '+1h 23:00' + '+1h 23:00', …).
Root cause: during the rolling `.ntms` → `.prop` rename window,
both extensions live on NFS for the same valid_time. `list_score_files`
matches either, so `list_valid_times_raw` mapped 2 files → 2 DateTimes
with no dedupe. Propagation.available_valid_times inherited the dup,
push_timeline pushed it to the JS hook, renderTimeline drew one chip
per entry — hence the visible duplicate.
Fix: `Enum.uniq` before the sort. Regression test writes a `.prop`
file, copies it as the same valid_time's `.ntms` legacy twin, and
asserts list_valid_times returns a single DateTime.
Also land a small refactor that was queued behind this: move the three
pure mechanism display helpers (label/1, badge_class/1, explainer/1)
out of the 2500-line contact_live/show.ex into a new
MicrowavepropWeb.ContactLive.Mechanism module. No behavior change — just
shrinks the LiveView and gives the display mapping a testable home.
- Add one-line moduledocs to 18 LiveViews that were carrying
@moduledoc false. Each line names the route and summarizes what
the page does so a future reader knows where to look before
opening the file.
- Add MapLive handle_event tests for toggle_radar, select_time /
set_selected_time, point_detail, map_bounds, retry_initial_scores.
Assertions focus on socket-state transitions and "didn't crash"
rather than raw HTML — map_bounds in particular has no visible
render-side effect (it push_events the new scores to the JS hook).
- Rust pipeline: integration-shaped test that hand-builds a 2-cell
surface + pressure grid, runs merge → cell_to_conditions →
precompute_band_invariants → composite_score_with → write_atomic
→ decode, asserting header + body length round-trip. Closes the
'pipeline's happy path is only covered by unit tests' gap.
StatusLive had a hand-rolled thousands-separator format_number/1 and a
binary-prefix format_bytes/1 — duplication that was already part of the
review punch-list. Move both to Microwaveprop.Format (which previously
held only distance_km) and delete the local defps. Fourteen call sites
switched to Format.number / Format.bytes.
Also extend the IEM error classifier to snooze on Req.TransportError
{:closed, :timeout, :econnrefused, :nxdomain}. Same rationale as the
429 snooze in batch 1: upstream TCP resets mid-fetch aren't a job
failure, they're transient — returning {:error, _} blows them up into
Oban.PerformError logspam with full stacktraces.
Batched from a system-wide review pass.
**Rate-limit + transient-error log hygiene**
- WeatherFetchWorker: classify IEM HTTP 429 as {:snooze, 300} instead
of {:error}. Stops Oban.PerformError stack-trace spam on every
routine rate-limit during backfill, keeps the retry semantics.
- IonosphereFetchWorker: compress GIRO TLS :unknown_ca error from a
~400-char inspect blob to 'TLS unknown_ca (CA bundle missing)'.
- FreshnessMonitor: log only when an enqueue actually lands (the
Oban unique conflict case was silently dropping jobs but still
producing 'enqueuing grid worker' info lines every 5 minutes).
**Perf**
- Rust grid_level_keys(): pre-format 'TMP:{p} mb' / DPT / HGT keys
once via OnceLock instead of format!()'ing per-cell. Removes ~3.6M
String allocations per f01..f18 chain step across pipeline.rs,
hrrr_points.rs. Same for the wgrib2 :(TMP|DPT|HGT): pattern in
hrrr_points.process_batch (sfc_pattern / prs_pattern OnceLock).
- MapLive preload_forecast: Task.async_stream with max_concurrency=4
replaces the 18-wide Enum.map serial walk. Forecast cache warms
~4× faster after a band change, with ordering preserved.
- grid_tasks: partial composite index on (run_time DESC,
forecast_hour ASC) WHERE status='queued' AND kind='forecast',
matching the Rust claim query's ORDER BY. Drops the old
status-only partial that forced a sort per claim.
**Correctness**
- ContactImportWorker: add Oban unique:[keys: [:import_run_id,
:offset]]. Was missing on a worker whose perform() does a
non-idempotent atomic counter increment — a retry would double-
count imported rows.
- CommonVolumeRadarWorker: x_min..x_max default step is -1 when
x_min > x_max, triggering a runtime deprecation warning. Force
Range.new(_, _, 1) explicitly.
**Cleanup**
- Drop unused tmp_dir parameter from hrrr_point_worker + its
process_batch signature.
Two unrelated production warnings from the same log window:
WeatherFetchWorker raised Postgrex 21000 cardinality_violation on
ASOS upserts when IEM returned two METAR records for the same
timestamp (routine + special at the same minute). Dedupe by
(station_id, observed_at) in upsert_surface_observations, keeping
the last occurrence since IEM's ordering is chronological.
SwpcClient.fetch_xrays warned with the full truncated response
body (~650 KB of GOES X-ray JSON) dumped into the log aggregator
whenever SWPC's CDN cut the response mid-stream. Intercept
Jason.DecodeError in fetch_and_parse and format it short: keep
the byte position and an 80-char preview, drop the rest.
/live fires every 10s per pod; the request line and Sent 200 response
from Plug.Telemetry drowned out real requests. log_level/1 now returns
false for /live too, same treatment /health already got.
The forgejo-runner was offline when 989d310..410a137 landed, so the
build-grid-rs workflow never dispatched for those pushes. Empty CI
retrigger commits don't match the path filter. Touching the crate
root forces build-grid-rs.yaml to fire and pick up the HRRR S3
fallback + scores_file .prop rename + point lookup fix.
Also corrects the now-stale "f01..f18 forecast hours only" note — as
of Phase 3 Stream A, Rust owns the f00 analysis step too.
Investigated why all three prop pods were restart-looping every few
minutes (RESTARTS counts 7–11 in 99m). The trail:
20:53:33.058 [error] Postgrex.Protocol ("db_conn_10") disconnected:
client (:"Elixir.Microwaveprop.PromEx.Poller.5000") timed out
because it queued and checked out the connection for longer than
15000ms
PromEx's Oban queue-depth poller was scanning the oban_jobs table
every 5s and holding an Ecto connection past its checkout timeout.
That starved the rest of the pool, so /health's `SELECT 1` couldn't
acquire a connection inside the 3s liveness probe timeout, the
kubelet's failureThreshold=3 tripped, and SIGKILL landed on the
BEAM. Every replica cycled on the same cadence.
Two fixes:
1. Split /live (no DB) from /health (DB ping). Kubernetes livenessProbe
now points at /live so a saturated Ecto pool can never cascade into
SIGKILL. Readiness still hits /health so a genuine DB outage drains
the pod from Service endpoints gracefully.
2. Turn off the PromEx Oban plugin in prod via the same flag already
used in test. The queue-depth query isn't worth pod instability;
the oban_web dashboard surfaces the same information without
scanning the job table on a timer.
Locked both down with HealthController tests that verify /live never
touches the Repo (no sandbox owner, controller.live/2 called directly,
200 ok) and /health does (sanity query round-trips).
Two fixes cover the blank "analysis breakdown" panel the user reported:
1. Bound the point_detail fallback lookback to 24h. Previously
factors_from_fallback_profile would happily use a week-old analysis
profile when no current one existed — now anything older than 24h
returns an empty factor map so the UI can surface "breakdown
unavailable" instead of silently misattributing.
2. Move the Plausible analytics snippet from Layouts.app into
root.html.heex. Full-bleed LiveViews (/map, /contacts/map,
/weather, home) bypass Layouts.app, so the snippet only loaded on
the navbar-wrapped pages. root.html.heex is loaded once per HTTP
document so coverage is now universal.
Added ~30 tests locking both down:
• point_detail fallback: exact-time wins, 24h boundary accepted,
30h-stale rejected, multi-band coverage, missing-cell degrades to
empty factors, default-valid_time path
• analytics_test.exs: Plausible script present on 12 representative
pages, exactly once, loaded async, surviving a login round-trip
Also fixed pre-existing credo issues per standing rule: shortened the
map_live_test ETS match_delete call, extracted a function from the
deeply-nested hrrr_point_enqueuer.enqueue_for_contacts/1 cond, and
swapped Mix.Tasks.Rust.Golden's IO.inspect for Mix.shell().info.
Score-grid files now land at `<band>/<iso>.prop` with a 4-byte "PROP"
magic header. Readers still accept the legacy `.ntms` extension and
"NTMS" magic so a rolling deploy doesn't invalidate any file already
on the shared NFS mount — legacy files age out through normal
retention.
Applied both sides of the seam: Elixir ScoresFile + regex parsers,
Rust scores_file writer + decoder. Updated the comments in all
modules that mention the extension (NotifyListener, Reconciler,
gefs_fetch_worker, map_live) and the runbook diagram. talos5 is a
DB host now, not a Rust worker — corrected the runbook accordingly.
Root cause of the blank /map analysis breakdown: the skippy.w5isp.com
caching proxy was unreachable from the cluster, so every f00 analysis
step failed at the idx fetch and Rust never wrote a ProfilesFile. The
/map point-detail panel silently degraded to empty factors because
no profile ever lands on `/data/scores/profiles/`.
HrrrClient now takes an optional fallback base. When the primary base
exhausts its retries (5× exponential backoff), fetches retry once
against the fallback — defaulted to the NOAA S3 public bucket. Forecast,
analysis, and native-duct paths all go through the same
fetch_blob_with_fallback helper so a proxy outage degrades to direct
S3 instead of stalling.
Moved the native-URL builder out of native_duct into fetcher so the
single helper can build both primary and fallback URLs for the native
hybrid-sigma product.
Convert every context/worker upsert that used the catch-all
`:replace_all_except` form to the explicit `{:replace, [:col1, ...]}`
form. Reviewers can now see exactly which columns an upsert overwrites,
and adding a new column to a schema no longer silently opts it into the
update path.
Behaviour is preserved bit-for-bit: each new explicit list contains
every column the old `:replace_all_except` would have overwritten.
Touched:
- Microwaveprop.SpaceWeather (Kp / F10.7 / X-ray shared helper)
- Microwaveprop.Ionosphere (observation upsert)
- NexradWorker, HrrrNativeGridWorker (insert_all grids)
- CommonVolumeRadarWorker, RadarFrameWorker (per-contact stats)
Also pins the conflict behaviour for the ContactCommonVolumeRadar
upsert path in RadarFrameWorkerTest so a future column addition that
isn't reflected in the explicit list fails loudly.
Wrap a Task.Supervisor in a PartitionSupervisor (one partition per
scheduler) and route the four sustained async_stream callers through
`{:via, PartitionSupervisor, {Microwaveprop.TaskSupervisor, self()}}`.
Concurrent work — the hourly PropagationGridWorker pulling f00-f18
HRRR range downloads, the GEFS worker scoring its grid, and the
Recalibrator's 20-way positive/negative fan-out — no longer funnels
through a single supervisor PID.
Light/one-shot Task.async + Task.start sites (hrrr_client surface/
pressure fan-out, application boot warmup, weather fire-and-forget)
are left alone; partitioning only helps under sustained concurrency.
Two-stage mount splits the propagation map's first paint: the static
HTTP render still fetches scores synchronously so SEO/noscript shipping
gets real data baked into `data-scores`, but the websocket-connected
mount defers the potentially-slow fetch to `start_async/3`. Chrome
paints immediately; scores stream in via an `update_scores` push_event
once `handle_async(:initial_scores, ...)` resolves.
A small overlay (spinner on `:loading`, retry button on `:failed`)
uses `<.async_result>` so users can see state and recover from a
transient fetch error.
The ingress + Grafana pipeline parses JSON-formatted log lines; Elixir's
default text formatter was the odd format everywhere else, forcing Loki
to fall back to free-text parsing. Swap the `:default` :logger handler's
formatter to `LoggerJSON.Formatters.Basic` when `LOG_FORMAT=json`.
Defaults: `LOG_FORMAT=text` for dev/test (no behaviour change), `json`
for prod. Whitelisted metadata covers the fields we actually correlate
on — `request_id`, `trace_id`, `worker`, `queue`, `job_id`.
OTP 28 / Elixir 1.19 exposes `Logger.Formatter.new/1` but ships no
JSON encoder of its own, so `logger_json` (~> 7.0) carries the rendering.
The test pokes the formatter directly rather than round-tripping through
ExUnit's capture_log, which replaces the handler and so can't see the
installed formatter's output.
PromEx.Plugins.Oban counts job exceptions as a Prometheus counter but
does not tell us which worker/queue failed, whether the job is about to
be retried, or the exception class. This adds a telemetry handler on
[:oban, :job, :exception] that:
- emits a single structured Logger.error with worker, queue, job_id,
attempt, max_attempts, retry_exhausted?, kind, reason class, and
args-keys summary (never values — args can contain PII/tokens);
- dispatches opt-in per-worker recovery callbacks
(on_permanent_failure/1 when retries exhausted,
on_transient_failure/1 otherwise);
- catches callback failures so a bug in one worker's recovery path
cannot detach the handler or break logging for other jobs;
- does not emit a second telemetry event, so PromEx counts are not
double-counted.
Handler is attached from Application.start/2 after the Oban supervisor
starts.
Every pod was rescheduling its sweep on a fixed 60 s `Process.send_after`
tick, so N replicas hit the shared NFS `/data/scores` mount and Postgres
in lock-step every minute. Add a uniform jitter of up to 20 s on top of
the 60 s base (effective range 60-80 s) so per-replica schedules drift
apart.
Expose `next_sweep_interval/0,2` as a public helper so the randomized
window can be tested deterministically with `:rand` draws rather than
mocking timers. Existing reschedule test passes `jitter_max_ms: 0` to
keep its 2 s wait window honest.
Previously the `mix deps.patch` task lived at `lib/mix/tasks/deps.patch.ex`
and the `deps.get` alias invoked it. This broke `mix deps.get --only prod`
in Docker: `lib/` isn't compiled yet when deps are fetched, so the task
module couldn't be resolved and the build failed with
`The task "deps.patch" could not be found`.
Move the patch-application function directly into `mix.exs` as a private
helper and wire it into the alias via `&apply_dep_patches/1`. `mix.exs` is
always loadable, so the function is available the moment `deps.get` runs.
Cover the pure-math invariants of the ITU-R P.526-16 diffraction
helpers, the haversine metric on Geo, and the Magnus / dry-adiabat
helpers in the skew-T renderer. 29 properties, one invariant each.
- HrrrClient idx-cache test only invalidated the surface idx URL, but
fetch_profile also fetches a pressure idx. Previous runs' state for
the pressure key decided whether the counter landed at 1 (prior run
cached it, pass) or 2 (cold, fail). Invalidate both + assert 2 total
fetches to reflect the actual code path.
- CsvImportTest deadlocked against other async DataCase tests when
inline Oban child jobs upserted iemre_observations/terrain_profiles
with a shared conflict target. Flip to async: false — same fix as
ContactWeatherEnqueueWorkerTest earlier this session.
- PromEx.Plugins.Oban runs a 5s telemetry_poller that queries the DB,
but its poller PID has no sandbox connection in test and crashed
with DBConnection.OwnershipError on every tick, spamming the log.
Gate the plugin on a config flag and skip it in config/test.exs;
prod behaviour unchanged.
Band-warm failures in the Rust propagation_ready pipeline were only
surfaced via Logger.warning, so Prometheus had no way to see them.
The /weather GridCache hit/miss was likewise invisible while its
/map counterpart (ScoreCache) already reported cache ratio.
- NotifyListener emits [:microwaveprop, :propagation, :notify_listener, :warm]
with %{ok, err} measurements and %{valid_time} metadata after every
propagation_ready notification. Public handle_propagation_ready/1 so
tests can exercise the warm+telemetry path without Postgrex.
- GridCache.fetch/1, fetch_bounds/2, and fetch_point/3 emit
[:microwaveprop, :weather, :grid_cache, :lookup] with a :hit | :miss
metadata key on every ETS lookup.
Wraps the hourly grid_tasks seed with
[:microwaveprop, :propagation, :grid_worker, :perform] so Prometheus
can see duration, success, and failure for the chain-seed cron.
Emits a :seeded event with the queued_tasks count and an :exception
event on GridTaskEnqueuer failure so silent seed errors show up in
the existing PromEx dashboards instead of dying as a lone Logger.info.
Set OTEL_EXPORTER_OTLP_ENDPOINT on the prop deployment to the cluster
Collector so Phoenix/Bandit/Oban spans export and link up with
downstream prop-grid-rs / prop-hrrr-point-rs traces in Tempo. Without
this the Elixir app fell back to traces_exporter :none and the Rust
worker spans had no parent, leaving the trace graph disconnected.
The stale comment referenced opentelemetry_ecto as the reason export
was off, but that package was never added to mix.exs (only
opentelemetry_phoenix / _bandit / _oban are listed), so the
span-flood concern no longer applies.
Weather.find_nearest_hrrr/3 and Weather.find_nearest_native_profile/3
run a three-range scan gated first by a ±1h valid_time window, then
by a ±0.07° lat/lon box. The existing unique `(lat, lon, valid_time)`
index is only usable on its `lat` prefix for these queries, forcing
Postgres to scan wide lat slices before applying the valid_time
filter — expensive on the Turing Pi 2 node.
Add a `(valid_time, lat, lon)` composite so valid_time leads,
collapsing the search to a narrow btree slice inside a single
partition before the lat/lon box filter kicks in.
`hrrr_profiles` is RANGE-partitioned by valid_time, so
CREATE INDEX CONCURRENTLY is applied per partition and the resulting
indexes are attached to a parent index created ON ONLY the partitioned
table. `hrrr_native_profiles` is a regular table, so the concurrent
build runs directly on it.
IemClient.fetch_asos returns 24-288 rows per station per call, each
previously triggering an individual UPDATE-conflict round-trip via
Weather.upsert_surface_observation/2. On the Turing Pi 2 Postgres
node that per-row latency dominates ingestion time.
Add Weather.upsert_surface_observations/2, a bulk variant that collapses
the rows into a single Repo.insert_all with the same
update-only-when-changed on_conflict predicate and (station_id,
observed_at) conflict target. Switch WeatherFetchWorker to use it.
The remote_ip plug previously trusted Cf-Connecting-Ip and X-Forwarded-For
on every request, so any client reaching a pod directly (cluster-internal,
kubectl port-forward, misconfigured Service) could spoof conn.remote_ip,
which flows into session storage and logs.
Now only honour those headers when the immediate TCP peer sits inside a
configured CIDR list. Default list covers loopback, RFC1918, CGNAT,
link-local, and IPv6 ULA — matches the k8s pod/service network.
Override via config :microwaveprop, :trusted_proxies, or per-environment
via the TRUSTED_PROXY_CIDRS env var in runtime.exs.
Also refresh the stale "nginx/dokku proxy" moduledoc — deployment has
been k8s + Cloudflare for a while.
The Microwaveprop.Weather.GridCache ETS table grew unbounded — each
hourly PropagationGridWorker run plus peer broadcast_put added ~92k
points per forecast hour, so long-lived pods were strong OOM
candidates. GridCache already exposed prune_older_than/1 but nothing
called it.
Adds an Oban worker on the :propagation queue (lower priority than
the grid chain) that prunes entries older than 3 hours. Scheduled
hourly at :15 in dev, prod, and the base crontab.
CommonVolumeRadarWorker and MechanismClassifyWorker were both
returning :ok on upstream failures, so Prometheus job-failure counters
stayed at zero during multi-hour NEXRAD outages and classifier bugs
went unnoticed.
- CommonVolumeRadarWorker: distinguish permanent (4xx) from transient
errors using the same pattern as RadarFrameWorker. Permanent errors
still mark :unavailable + :ok so the backfill stops re-queueing them.
Transient errors return {:error, reason} and leave radar_status alone
so the next retry can succeed.
- MechanismClassifyWorker: keep the row-status :failed update so
operators can see which contacts blew up, but return
{:error, inspect(e)} from the rescue instead of swallowing it as :ok.
- MapLiveTest timestamp indicator: earlier tests in the suite leave
score files in the shared `:propagation_scores_dir` and warm the 5 s
`list_valid_times` cache; clear both in setup so the "No data"
assertion sees the state the test documents.
- PropagationPruneWorkerTest: `old = now - 3h` collides with the
worker's cutoff (also `now - 3h`) once both are truncated to the
second, so they land equal when test + worker run in the same wall
clock second. Shift `old` by an extra -1s for unconditional ordering.
- ContactWeatherEnqueueWorkerTest: flip to async: false — perform/1
runs inline Oban child jobs that upsert rows with a shared conflict
target, and parallel DataCase siblings hit "ShareLock deadlock_detected"
intermittently.
