Each /weather/tiles request was running the full ScalarFile decode path
(File.read → gunzip → Msgpax.unpack → normalize) for any non-analysis
valid_time, because GridCache deliberately skipped forecast hours to keep
memory low. With 21 simultaneous viewport tiles all hitting the same few
chunk files, each tile took 1.5-3.2s of contended IO + CPU.
On a GridCache miss when a ScalarFile exists, hydrate GridCache from the
file once (deduped via the existing claim_fill primitive) and serve all
subsequent reads from ETS. Bound memory with prune_keep_latest/1 — keep
the 24 most-recently-touched valid_times, which covers a full HRRR run
(analysis + 18 forecasts) plus a few stragglers from the previous run.
In the steady state, the first viewport read for a given forecast hour
takes one full ScalarFile decode (~50-100 ms for 92k cells); every
subsequent tile, point lookup, and viewport read for the same hour
serves from ETS in <1 ms.
Move the /weather render path off raw HRRR profile decoding + per-cell
SoundingParams + WeatherLayers derivation. The dominant cost was
re-deriving the same scalar fields on every viewport pan and timeline
scrub.
Server side:
- New Microwaveprop.Weather.ScalarFile persists pre-derived scalar
rows on NFS, bucketed into 5°×5° chunk files under
<base>/weather_scalars/<iso>/<lat_band>_<lon_band>.etf.gz. Viewport
reads only decode the chunks that overlap the requested bounds;
point-detail clicks read exactly one chunk.
- weather_grid_at/2 and weather_point_detail/3 prefer ScalarFile;
ProfilesFile is now the cold-start fallback only.
- warm_grid_cache_and_broadcast/1 and warm_grid_cache_from_latest_profile/0
also persist the scalar artifact, and the cold-derive path kicks
off a per-valid_time-locked async materialization so the next
reader gets the cheap path.
- NotifyListener.handle_propagation_ready/1 fires
Weather.materialize_scalar_file/1 in a detached Task on the Rust
pipeline's NOTIFY propagation_ready, so steady-state forecast hours
arrive with their scalar artifact already on disk.
- available_weather_valid_times/0 unions ScalarFile + ProfilesFile so
the timeline survives an aggressive retention sweep on either side.
Browser side (finding #8):
- Mount the legend Leaflet control once and patch its inner content
on layer changes instead of remove + re-add on every renderLayer.
- renderTimeline now keys off the rendered button set: selection-only
updates take an applyTimelineSelection patch path that restyles
existing buttons in place. Full innerHTML rebuild only fires when
timelineData itself changes.
Also fixes a credo nesting-depth flag in tile_renderer.ex by extracting
interpolate_pair/2 from the inner anonymous function.
Fixes flagged by the code-reviewer agent's pass over the session's
commits (cc9220b..7b78a25):
- Propagation.warm_cache_and_broadcast/2 now uses ScoresFile.read/2
directly and returns {:ok, :ok} | {:error, :enoent | :invalid_format}.
Previously it called ScoresFile.read_bounds/3 which silently returns
[] on missing/corrupt files, poisoning ScoreCache with an empty grid
that the reconciler couldn't heal.
- NotifyListener.warm_band/2 and ScoreCacheReconciler.warm_one/2 now
pattern-match {:error, reason} and skip (log, return :error) instead
of caching empty. rescue clauses kept as defense-in-depth for
unexpected faults in PubSub.broadcast / ETS writes.
- ScoresFile.extract_points/2 promoted to @doc public — callers that
need to distinguish missing file from empty grid can feed read/2
payloads here themselves.
- Weather.build_grid_cache_row/4: replaced || fallbacks with prefer/3
helper (Map.fetch) so a legitimate persisted ducting_detected: false
is not clobbered by a derived-from-sounding true.
- Weather.hrrr_data_fully_present?/1 @spec tightened from map() to
Contact.t() | field-constrained map, plus is_nil(qso_timestamp)
guard so callers with partial contacts get a clean false rather
than a HrrrClient.nearest_hrrr_hour/1 crash.
- AdminTaskWorker.native_derive bulk-UPDATE: chunk reduced 2000 → 500
and wrapped in try/rescue with per-row fallback on Postgrex errors
so a single bad row doesn't kill the remaining 1999 in its chunk.
- Runbook FM3 rewritten to match the actual code path (no rescue;
explicit {:error, _} pattern match). FM5 clarifies the surviving
PropagationGridWorker is a cron-fired seed worker, not a fallback
compute path.
- Dialyzer: strict flags added (:error_handling, :unknown,
:unmatched_returns, :extra_return, :missing_return). Baseline
emitted 130 warnings, mostly discarded Task.start/PubSub/Logger
returns; a follow-up will tighten those and backfill @specs.
New tests:
- ScoreCacheReconciler GenServer lifecycle: run_on_start true/false,
interval_ms rescheduling, info-level log line.
- Weather.hrrr_data_fully_present?/1: nil qso_timestamp returns false.
- Weather.build_grid_cache_rows/2: explicit ducting_detected: false
on profile beats derived true from sounding params.
- RadarFrameWorker: pins the NexradClient "NEXRAD n0q HTTP <code>"
error string contract so permanent_error?/1 classification doesn't
silently regress if the client's error format changes.
- AdminTaskWorker: remove scorer_diff no-op clause (propagation_scores table
was dropped pre-cutover; queue is drained).
- Weather: remove load_weather_grid_from_db/1 fallback — Rust writes
ProfilesFile for every forecast hour, so the DB path was unreachable
in practice and loaded stale pre-cutover data when it fired.
- Weather.build_grid_cache_row/4 now prefers persisted fields from the
ProfilesFile profile map (surface_refractivity, min_refractivity_gradient,
ducting_detected, duct_characteristics) over re-deriving from the raw
profile, matching the old DB-path semantics.
- NotifyListener: wrap per-band warm in try/rescue so one corrupt or
mid-rename .ntms file doesn't abort the remaining bands. Failures log
a warning with band + valid_time; successes still emit one info line.
- weather_grid_test: rewrite insert_hrrr_profile helper to write
ProfilesFile directly and drop the "only returns points on 0.125 grid"
test (filter was a property of the deleted DB fallback; Rust writes
only grid points by construction).
The Contact list's 'Partial' badge was sticky: once a worker gave up on
a contact (upstream 404, no nearby station, IEM throttling) or the
stale-queued reconciler flipped it to :unavailable after 3 days, the
backfill cron never looked at it again. Upstream archives heal over
time — IEM backfills gaps, new ASOS stations come online, Rust worker
gains capability — so rows stayed 'Partial' that shouldn't have.
Changes:
* BackfillEnqueueWorker.type_filter now also matches <status>_status =
:unavailable when contacts.updated_at is older than a 24 h cooldown.
enqueue_for_contact is idempotent: if there's still nothing to
fetch it'll flip the status to :complete on the empty-jobs branch;
if new data shows up it builds fresh jobs.
* reconcile_stale_queued_to_unavailable now bumps updated_at when
flipping to :unavailable so the cooldown starts fresh — otherwise
the main loop would pick it right back up on the same cron tick
(stale updated_at < any reasonable cooldown) and immediately
transition :unavailable → :complete, defeating the :unavailable
signal entirely.
Also:
* Drop ScoresFile.point_score/3, dead since read_point moved to the
pread fast path in commit bd3b114.
* Clear Microwaveprop.Cache in WeatherGridTest setup. The new
ProfilesFile read cache is keyed by (base_dir, valid_time); tests
in this file reuse DateTime.utc_now() truncated to seconds and
share the default base_dir, so a prior test's cached read leaks
into the next one's expectations.
Timeline scrubs on /weather were reading a 21 MB NFS ProfilesFile and
then calling SoundingParams.derive on all 92k CONUS points before
filtering to the viewport. Flip the order: filter the raw grid_data map
by bounds first, so derivation work is bounded by the viewport (~4k
points on a DFW box) instead of the full grid.
Add build_grid_cache_rows/3 with an optional bounds argument; the /2
arity is unchanged so PropagationGridWorker still derives the full grid
on f00.
Extend the f00 profile persistence to cover every forecast hour
(f00-f18) so /weather keeps working across pod restarts and shows
forecast-hour atmospheric data without a fresh database. Route the
Weather cold path (latest_grid_valid_time, load_weather_grid,
weather_point_detail) through ProfilesFile first, with the legacy
hrrr_profiles table as a last-resort historical fallback. Warm
GridCache from the latest profile on app start so /weather is hot
the moment a pod boots.
warm_grid_cache_and_broadcast was re-SELECTing 92k hrrr_profiles rows
with JSONB profile/duct_characteristics columns on every forecast hour.
Row decoding exceeded the 15s default DB connection timeout, killing
the worker before f01-f18 could run. Oban retried from f00 and got
stuck in a loop: for the last several hours, no forecast hours were
being written and even f00 was stale.
Build the weather cache rows directly from the in-memory grid_data
the worker already has (Weather.build_grid_cache_rows/2) and
GridCache.broadcast_put directly. No DB round trip, no JSONB decode.
Also widen the cron from hourly to every 3 hours so a full f00-f18
sweep (~95 min) can actually complete before the next run starts,
and drop the redundant prune_old_scores() at worker start
(PropagationPruneWorker already runs every 15 min). Add a 60s query
timeout to load_weather_grid_from_db as defense-in-depth for the
cold-cache fill path that still uses it.
The /weather LiveView was timing out on pod restart because
latest_weather_grid/1 ran load_weather_grid_from_db/1 synchronously on
cache miss — that query reads 176k hrrr_profiles rows with two huge
JSONB columns (profile array + duct_characteristics), and JSONB
decoding blocked the LiveView process past the 15-second pool
ownership timeout.
On cache miss latest_weather_grid/1 now returns empty immediately and
kicks off a deduped background fill through GridCache.claim_fill/1 (a
per-valid_time ETS lock so N concurrent mounts after restart don't each
fire the slow query). When the fill completes it broadcasts
weather:updated, and the handle_info in WeatherMapLive hits the warm
cache and pushes rows over the socket.
Added load_weather_grid/1 as a synchronous sibling for tests and any
caller that genuinely needs inline data. Tests updated to use it and
to clear GridCache in setup so ETS state doesn't leak between cases.
Also untrack k8s/secret.yaml and add both it and k8s/*-secret.yaml to
.gitignore — those manifests hold plaintext SMTP and database
credentials that should not live in the repo. Secrets already in git
history should be rotated separately.
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
