When wall-clock is past :30, the hour *after* Now rounded to "0h" and
the Now slot kept its "Now" tag — two adjacent buttons both reading
as the current hour. Anchor the offset math to the Now slot itself so
the label sequence reads "-1h / Now / +1h / +2h" regardless of where
inside the hour the user loaded the page.
The host docker socket wasn't mounted consistently across act_runners,
so which runner picked up the push was the difference between a clean
build and "unable to connect to docker." Switch to a dockerd service
pinned to the same bridge network as the job; the CLI now talks to
it over TCP via DOCKER_HOST=tcp://dockerd:2375. Every runner has a
daemon, every time — no more re-runs for the scheduling lottery.
The 30s poll is enough to ride out a slow daemon startup, but not a
runner where the socket isn't mounted at all. Print DOCKER_HOST, the
socket stat, and the verbose `docker info` error on timeout so the
next flaky run tells us whether it's a startup race or a missing
mount.
Forgejo act_runner mounts the host docker socket into the job
container; the daemon sometimes takes a beat to answer, which has
been surfacing as intermittent "unable to connect to docker" failures
on login. Gate the build on a `docker info` poll (up to 30s) so later
steps don't race the socket, and wrap login + buildx push in a 3-try
exponential-backoff (2s/4s/8s) so a transient registry hiccup doesn't
require a manual workflow re-run.
Three UI fixes:
1. MapLive now ticks every 60s and, when the selected_time is still
the "Now" slot, advances to the newer latest-past-or-now hour as
the clock crosses into it. The URL is patched in place so shared
links stay accurate without polluting history. Manually picking a
specific past/future hour turns tracking off.
2. point_forecast/3 now reads its valid_times list straight from the
on-disk .ntms files (same source the main-map timeline uses), then
consults the cache per-hour for a fast score lookup. The old cache-
or-store branch could leave the click-to-detail sparkline at 13
hours while the bottom timeline showed 14+; both now line up.
3. Point-detail panel moves from bottom-14 → bottom-24 on desktop so
its lower edge clears the forecast-timeline bar (+ max-height tuned
to match), fixing the overlap visible in screenshots.
Four changes sized by measured prod telemetry (83m of spans):
1. propagation queue: 2 → 1 slot per pod. Two concurrent forecast-hour
steps per pod stacked HRRR grid + native duct grid + scored band
map into ~5-6 GiB RSS, OOM-killing every ~15 min. 3-way parallelism
cluster-wide still finishes the chain inside the hourly interval.
2. weather queue: 3 → 1 slot per pod. ASOS backfill was 429-thrashing
IEM (1,296 retryable jobs; logs were nothing but 429 backoffs).
3. PropagationGridWorker: skip native-level duct fetch on f01..f18.
At ~7-11 min/fh and 18 forecast hours, this was the largest single
cost per chain. Forecast hours fall back to
derived[:min_refractivity_gradient] from the pressure-level
profile. f00 still gets full native-level duct analysis.
4. HrrrClient.download_grib_ranges_to_file: parallelize with
Task.async_stream (max_concurrency 8). The file-backed variant was
sequential, dominating native-duct fetch time on the remaining f00
path. ~20s → ~3s per call.
The :score_band span was reading point_count via length(grid_data),
but grid_data is a %{{lat, lon} => profile} map. Every forecast-hour
job was crashing with ArgumentError before any scores landed, leaving
the map without current-hour forecasts. Swap to map_size/1.
Two wins on the hourly propagation pipeline:
1. Parallelize the chain. seed_chain was enqueuing only f00 and
each step self-enqueued f00+1, serializing ~2.5 min × 19
forecast hours into a ~48 min chain. Fan out all 19 jobs at
once — they're genuinely independent (different HRRR URLs,
different output files) — and let queue concurrency (2 slots
× 3 pods = 6 parallel workers) drop wall time to ~10 min.
Side effect: one permanently-failing step no longer takes out
the rest of the chain, so the rescue-chain logic I added last
commit becomes unnecessary. Removed.
The :final cleanup (retain_window + purge) used to run on the
fh=18 transition; with parallel execution we don't know which
step finishes last. Cleanup now relies on the existing
PropagationPruneWorker 15-min cron for time-based pruning.
Stale chain leftovers are already a minor concern with the
15-min prune; if file bloat becomes real, PruneWorker can
gain retain_window later.
2. Hoist band-invariant factors. score_time_of_day, score_sky,
score_wind, score_pressure depend on conditions only, not the
band — but composite_score was recomputing them 17 times per
point. Added Scorer.precompute_band_invariants/1, called once
per point before the bands loop; composite_score uses the
cached values when present and falls back to computing for
any caller that doesn't pre-warm (test harness, path
integrator). Saves ~30% of the scoring inner loop.
Telemetry showed ~66 PropagationGridWorker exceptions per 6h with
55 ArgumentErrors and 11 TimeoutErrors, producing ~13 discarded
chain steps. Each discard broke the chain: subsequent forecast
hours were never enqueued, leaving the score store with huge gaps
(e.g. at 14:11 UTC the earliest available forecast was 18:00,
because f00-f05 all failed somewhere upstream and nothing ran
after them).
Three changes:
1. PropagationGridWorker: on the final attempt, still enqueue
fh+1 even when this step failed. Oban discards the current
job normally — but the rest of the chain keeps running, so
one bad hour doesn't take out the remaining 12-18. The
rescue is factored into a tested public helper.
2. HrrrClient.parse_idx: skip malformed idx lines instead of
raising. NOAA S3 occasionally serves an HTML error page as
the idx body, and the old strict String.to_integer path
raised ArgumentError on the first non-numeric line and took
down the chain step. This is the root cause of the 55
ArgumentErrors.
3. JS renderTimeline: when no forecast hour is at-or-before
wall-clock (all times are future — the gap scenario the
fixes above are designed to prevent), stop labeling the
earliest future slot "Now". Lets the user see honest
"+Nh" offsets instead of a lie on the pill.
Telemetry puts nexrad_decode_png at 1.76 s/call × 37,847 calls/h
— ~18.5 h of CPU across the cluster every real hour, the single
biggest CPU consumer in the system. The map-click path
(fetch_rain_cells) already cached decoded frames by 5-min
bucket, but the per-contact CommonVolumeRadarWorker path
(fetch_decoded_frame) went straight to network + decode on every
call. Backfill means many contacts share a 5-min window, so the
same 66 MB frame was being decoded dozens of times.
Wire fetch_decoded_frame through NexradCache keyed on the
rounded timestamp. Add a 20-entry size cap in NexradCache so
backfill processing contacts in random timestamp order can't
grow the ETS table to hundreds of GB. Each frame is ~66 MB, so
20 = ~1.3 GB worst case, well under typical pod memory.
Expected impact: cuts sustained decode load by an order of
magnitude depending on backfill temporal locality; map-click
path is unchanged.
Every parser already returned nil on malformed input, so the ||
chains fell through to defaults — except parse_time_param, which
happily accepted any well-formed ISO8601 even if it pointed
outside the current HRRR forecast window (e.g. a stale shared
link from last week). Gate it behind the valid-times window and
fall back to the normal 'now' cursor when it's out of range.
Adds explicit coverage: non-numeric band, unknown band MHz,
non-numeric lat/lon/zoom, out-of-range values, malformed time,
ancient time, and all-params-garbage-at-once. Each case mounts
cleanly and lands on the defaults.
The main map page now reads band, selected time, and map
center/zoom from URL params and updates the URL as the user pans,
zooms, picks a band, or scrubs the timeline. Pan/zoom use
push_patch replace-mode so browser history isn't spammed, while
band and time changes push fresh entries.
Also fix the 'Up to date · just now ago' pipeline chip — 'just
now' already reads as an instant so the trailing 'ago' is
redundant. Any older duration keeps it.
The :propagation queue (2 slots) is shared by PropagationGridWorker,
MrmsFetchWorker (every 2 min), and PropagationPruneWorker (every
15 min). Oban dispatches priority-then-FIFO, so a post-deploy
MRMS/prune backlog could delay the hourly chain step until the
siblings drained.
Set PropagationGridWorker priority: 0 explicitly and drop the two
siblings to priority: 5 so new chain steps always jump ahead.
Backfill workers live on separate queues (:hrrr, :weather, :terrain,
:iemre, :narr, :radar, :mechanism) and already don't interact with
the predictor.
- Remove the 7-day outlook strip and daily_outlook_at/3. It was
derived from point_forecast, which is capped at HRRR's 18-hour
horizon, so it never had 7 days of data to show.
- factors_for: fall back to the nearest persisted analysis profile
at or before the requested time. Only f00 hours persist profiles,
so clicking any other hour used to yield an empty Analysis and
factor table. After the recent cursor fix lands 'Now' on a
forecast hour more often, this regressed further.
- available_valid_times/1: cap the forward horizon to 18h from now.
Leftover valid_times from prior cycles used to pile on the
timeline without adding information.
Both the Elixir initial-cursor pick and the JS timeline renderer
used nearest-by-absolute-distance, so any wall-clock time past
:30 snapped the 'Now' label onto the next top-of-hour forecast —
up to ~30 min in the future. Switch to the latest valid_time at
or before wall-clock, with a fallback to the earliest time when
every slot is future (shouldn't happen for HRRR, safe default).
Contacts can be marked private at submit time (single form, CSV
import, ADIF import) and edit time. Private contacts are hidden
from the public map, the public /u/callsign profile, and the
browse table for anonymous and non-owning viewers. The original
submitter and admins see them inline on the browse table with a
"Yes" badge, and the detail page shows a lock icon. Non-authorized
viewers get a 404 on the detail page.
Adds spans to 15 previously-unmeasured hot paths so every question we
might ask while tuning has a histogram to answer it:
External I/O:
- iem.fetch_iemre (gridded weather reanalysis)
- mrms.list_latest / mrms.download (precip radar)
- rtma.fetch_observation
- ncei.fetch_metar (historical 5-min METAR backfill)
- solar.fetch_indices (GFZ solar indices)
- swpc.fetch (SWPC Kp/F10.7/X-ray)
- giro.fetch (ionosonde)
- qrz.request, geocoder.geocode (callsign enrichment)
- srtm.download_tile (terrain tile download + gunzip)
- hrrr.download_grib_ranges (parallel byte-range fetch phase)
Subprocess:
- wgrib2.extract_grid / extract_grid_from_file / extract_grid_from_file_mapped
LiveView hot paths:
- propagation.scores_at (map score fetch + cache hit/miss counter)
- propagation.point_forecast (sparkline)
- propagation.point_detail (click-to-inspect)
- propagation.daily_outlook_at (/map outlook strip)
Worker-level end-to-end:
- worker.terrain_profile
- worker.mechanism_classify
- worker.mrms_fetch
Each event is registered in Microwaveprop.PromEx.InstrumentPlugin as
a Prometheus histogram (default / long buckets as appropriate) plus
a counter for the scores_at cache hit/miss ratio. Prometheus at
10.0.15.25 will start seeing the new series on the next scrape after
deploy.
A dedicated Prometheus at 10.0.15.25 on the private network can now
scrape the cluster via `<any-node-ip>:30090/metrics`. Scrape rotates
across pods; BEAM metrics stay representative as a sample and the
oban queue-depth gauge reads from the shared DB so all pods report
the same value.
PromEx.Plug's internal path check doesn't match when forwarded under
/metrics (conn.path_info is [] after the forward strips the prefix),
so it silently returned without sending a response. Call
PromEx.get_metrics/1 directly and send the text body ourselves —
tested, returns 200 with all 23 registered event handlers producing
proper Prometheus histogram output.
Wires PromEx with its built-in Application / BEAM / Phoenix / Ecto /
Oban plugins plus a custom `Microwaveprop.PromEx.InstrumentPlugin`
that registers Prometheus histograms for every
`Microwaveprop.Instrument` span (HRRR/NEXRAD/IEM/GEFS/NARR/UWYO
/Elevation fetches, DB batch upserts, terrain analysis, radar
aggregation, propagation score band). Queue-depth gauges from the
10-second poller land at `microwaveprop_oban_queue_count{queue,state}`.
Router forwards `/metrics` to MicrowavepropWeb.MetricsPlug which
optionally requires a bearer token (`PROMETHEUS_AUTH_TOKEN` env var in
runtime.exs); leave it unset to expose metrics publicly and restrict
at the ingress / Cloudflare Access layer instead.
Example external Prometheus scrape config:
scrape_configs:
- job_name: microwaveprop
scrape_interval: 30s
metrics_path: /metrics
scheme: https
authorization:
type: Bearer
credentials: <token>
static_configs:
- targets: ['prop.w5isp.com:443']
Previously the CommonVolumeRadarWorker ran silently — no URL logged,
no sign in the logs that it was doing any work. Added info-level
fetch/ok/error logs in NexradClient (same shape as WeatherFetchWorker
RAOB logs) plus start-of-work + ingestion lines in the worker itself.
Unchanged: the existing "no frame" warning and all control flow.
Nav bar and browser tab now show "Microwave Propagation" instead of
"NTMS Propagation Prediction". The About and Privacy pages still
reference NTMS since those describe the organization running the
service, which is unchanged.
Contact detail now searches for soundings in progressively wider radii
(150 → 300 → 600 → 1000 km) before declaring "No soundings found".
When even 1000 km turns up nothing, enqueue WeatherFetchWorker RAOB
jobs for every station at the widest non-empty radius. The worker
passes the contact_id through so it can PubSub back to
contact_enrichment:<id> as each sounding lands, and the LiveView
rehydrates without a refresh.
Also adds Weather.soundings_with_widening_radius/1 and
ContactWeatherEnqueueWorker.enqueue_raob_fetch_with_widening/1 so the
same logic can be reused from other callers.
Add Microwaveprop.Format.distance_km/1 and a matching formatDistanceKm
helper for the JS side. Both emit "X mi (Y km)" with one decimal under
10 mi and whole numbers above.
Replace the ad-hoc format_dist/format_km_mi helpers with the shared
formatter and update every user-facing distance render: contact
detail, contacts index column, user profile contact/involving tables,
path finder summary + sounding list, contacts map line popups, beacon
coverage tooltip, and propagation map range estimate + rain scatter
cell popup.
Add optional height1_ft/height2_ft to contact schema, submit form, and
edit form (direct + suggest-edit paths). Heights flow through to the
elevation-profile terrain analysis so clearance and diffraction are
computed from the actual antenna height instead of a 10-ft default.
Editing heights resets terrain_status and re-enqueues
TerrainProfileWorker so the stored path analysis picks up the new
geometry.
Also fix mark_likely_duct/3: the pattern was grabbing the element
instead of the index, so no duct ever got flagged as the likely
propagation path even when extract_ducts returned a valid layer.
The Dockerfile uses `--mount=type=cache` which requires BuildKit.
The static docker tarball only ships the CLI, so the plugin has to
be installed separately. Also flip the build step to
`docker buildx build --push` so build and push happen in one pass
instead of two (avoids the intermediate local-daemon load).
The runner image's bookworm apt repos have been failing GPG
verification with "At least one invalid signature was encountered"
on every job. Clearing lists and reinstalling debian-archive-keyring
on the host doesn't help because the runner spins up a fresh
container per job. Dodge apt entirely by grabbing the static Docker
27.5.1 client binary from download.docker.com. The binary talks to
the host's /var/run/docker.sock that act_runner mounts into the job
container, which is all we need for build + push.
