Flip the cron from every-3-hours to every-hour now that a full
f00-f18 chain runs in ~45-60 min instead of ~170 min. With the
:propagation queue's concurrency-of-2, a slow chain won't block
the next one — they interleave, and the file store's last-writer
-wins semantics give the newer run_time's analysis data naturally.
Add ScoresFile.retain_window/2 that deletes any file whose
valid_time falls outside [run_time, run_time + 18h]. Call it at
fh=18 of the chain so leftover files from the previous chain
(specifically the old f00 whose valid_time the new chain doesn't
overwrite because it advanced by one hour) are discarded
immediately instead of waiting for the 2h prune cron.
The step transition path is extracted to handle_step_transition/2
to keep the nesting under credo's limit and make the final-step
logic easier to read.
Read-side cutover for the binary scores store and a companion
cleanup that removes the biggest remaining DB write from the hot
path.
Propagation.scores_at/3, available_valid_times/1, latest_valid_time/0,
latest_valid_time/1, earliest_valid_time/1, point_detail/4, and
point_forecast/3 all now prefer ScoresFile and fall back to the
propagation_scores table when a file is missing. The map render
path reads from /data/scores first; Postgres stays as a safety
net while dual-write is on. point_detail still pulls factors from
Postgres (analysis-hour rows only) and coalesces nil to an empty
map so the JS popup iterates cleanly.
replace_scores/2 is now gated by a postgres_writes_enabled? flag
(runtime env MICROWAVEPROP_SCORES_POSTGRES=false, or the
:propagation_scores_postgres app env key) so the binary-only path
can be benchmarked locally without the DB insert. Default stays
true.
PropagationGridWorker no longer calls store_hrrr_profiles —
persisting 92k grid rows × 19 forecast hours of JSONB profiles
was ~12 min of wall time per chain for a table only
AsosAdjustmentWorker read from. Per-contact HRRR enrichment
through HrrrFetchWorker still writes its own (is_grid_point:
false) rows. AsosAdjustmentWorker is disabled in all three cron
configs since its data source is gone.
DataCase resets the scores tree between tests so per-test
ScoresFile writes don't leak across cases, and ScoresFileTest
switches to async: false because it mutates the global
:propagation_scores_dir env.
First step of the disk-backed scores migration from the DuckDB plan
doc. Ended up shipping the raw-binary variant instead of Parquet
because the data is disposable after ~2h — the ecosystem benefits
of Parquet only pay off for long-lived datasets, and the binary
path has zero new dependencies.
Microwaveprop.Propagation.ScoresFile writes one file per
(band_mhz, valid_time) tuple under the configured scores dir,
default /data/scores in prod and priv/dev_scores in dev. Layout is
a 33-byte header plus a dense n_rows × n_cols uint8 array (255 is
the no-data sentinel). The whole CONUS grid serializes to ~93 KB
per band, and writes use the temp-then-rename pattern so NFSv4
concurrent readers never see a partial file.
Propagation.replace_scores/2 now materializes the score stream
once and dual-writes: Postgres on the primary path, then one
ScoresFile per band as a best-effort follow-up (any file error is
logged but doesn't fail the DB write, so we can verify the file
path in prod before cutting readers over).
Propagation.prune_old_scores/0 also clears expired score files so
the existing 15-minute prune cron covers both storage layers.
Dev configuration points at priv/dev_scores/, added to .gitignore.
Test configuration points at a per-run tmp directory.
node3's NVMe SSD is the new shared-storage home. Mount the whole
/data export from 10.0.15.103 at /data inside the pod (instead of
the old skippy export of /data/srtm at /srtm) so there's room for
future shared stuff — Parquet score files, maybe a DuckDB database,
or whatever the next thing is — without re-exporting a new path.
SRTM tiles now live at /data/srtm rather than /srtm. Update the
srtm_tiles_dir runtime config to match, and fix the two doc
pointers (CLAUDE.md, duckdb plan) that referenced the old path.
Already applied to the cluster and rolled out; pods confirmed
running with tiles visible at /data/srtm.
A full f00-f18 sweep takes ~170 min of wall time, longer than the
~90 min gap between deploys on this cluster. Over the last 7 days
every run was killed mid-sweep and zero runs completed. The
propagation_scores table only ever held the scraps from partial
runs that landed before the pod died, which is why the map looked
like it was showing "the current hour" (or nothing).
The worker now processes exactly one forecast hour per perform/1
(~8-10 min) and enqueues the next hour as a fresh Oban job. A cron
fire with empty args seeds the chain at f00; subsequent runs carry
run_time + forecast_hour args. At f18 the chain stops and the
pruner runs. Each step broadcasts propagation:updated immediately
so new hours appear on the map as they land.
Wall time per perform drops from ~170 min to ~10 min, so Lifeline's
2h rescue window is no longer a factor and a pod restart loses at
most one forecast hour instead of the whole sweep. Retries and
max_attempts now describe a single fh, not the whole chain.
Also: bump the :propagation queue from 1 to 2 slots so
PropagationPruneWorker can run alongside the chain job, drop the
worker timeout/1 from 90 min to 20 min to match single-fh runs,
and pull Lifeline rescue_after from 120 min to 45 min to keep the
safety net above the step timeout.
Adds a "Data from HH:MM UTC · Nh ago/now/+Nh" indicator above the
pipeline chip in both the mobile and desktop sidebars so the user
can tell which valid_time the map is showing when the bottom
timeline isn't visible.
The previous attempt to flip `-DUSE_PNG=ON -DUSE_AEC=ON` on wgrib2
broke the image build because wgrib2 v3.6.0's CMakeLists.txt
line 94 forces `find_package(g2c 1.9.0 REQUIRED)` whenever PNG or
JPEG2000 is enabled, and NCEPLIBS-g2c wasn't installed in the
wgrib2-builder stage.
Install NCEPLIBS-g2c v1.9.0 (the version wgrib2 asks for) in the
wgrib2-builder stage ahead of wgrib2 itself, with:
- USE_PNG=ON (needs libpng-dev, already present)
- USE_AEC=ON (needs libaec-dev, already present)
- USE_Jasper=OFF (default ON — would need libjasper-dev
and another runtime dep we don't need)
- USE_OpenJPEG=OFF (same rationale)
- BUILD_G2C=OFF (new experimental API, not needed here)
Then flip wgrib2's PNG/AEC flags back on. wgrib2's binary links
dynamically against libg2c.so so the final stage copies
/usr/local/lib/libg2c.so* from the builder and refreshes the
loader cache with ldconfig.
With the decoder restored, re-add MrmsFetchWorker to the prod
cron in runtime.exs. That worker pulls NOAA MRMS PrecipRate grids
every 2 minutes, which use GRIB2 DRT 5.41 (PNG-compressed) — the
whole reason we needed the PNG decoder in the first place. ASOS
nudges will now have a live radar overlay again.
The previous commit that added -DUSE_PNG=ON -DUSE_AEC=ON to the
wgrib2 cmake invocation broke the image build: wgrib2 3.6.0 turns
find_package(g2c 1.9.0 REQUIRED) on when those flags are enabled,
and NCEPLIBS-g2c isn't installed in the wgrib2-builder stage.
Revert to the stock wgrib2 cmake config so CI is green again and
pull MrmsFetchWorker out of the prod cron instead. Without the
PNG decoder the worker can't read MRMS PrecipRate (GRIB2 DRT
5.41) and was logging "packing type 41 not supported" every two
minutes. AsosAdjustmentWorker's MRMS path already handles an
empty cache gracefully — the ASOS nudge still runs, it just
drops the live radar overlay.
Re-add MrmsFetchWorker once the Dockerfile installs NCEPLIBS-g2c
and rebuilds wgrib2 with PNG/AEC support.
Oban's Lifeline plugin was rescuing PropagationGridWorker from
executing back to retryable every 10 minutes (runtime.exs) or
30 minutes (config.exs / dev.exs), but one full f00–f18 sweep
takes ~95 minutes and the worker's own timeout/1 is already
90 minutes. The rescue fired mid-run, the retry restarted at
f00, and after 3 rescues the job was discarded with errors: []
(Lifeline rescues don't write an error summary). prod's
propagation_scores never advanced beyond ~5 valid_times, so
the forecast timeline on /map only showed 3–4 hours instead of
the intended ~17.
Raise rescue_after to 120 minutes in all three configs — past
the worker's own 90-minute cap with headroom — and document
the contract so the next concurrency tweak doesn't regress it.
Also restore MrmsFetchWorker and AsosAdjustmentWorker to the
prod cron in runtime.exs: runtime.exs replaces config.exs's
plugin list wholesale and those two had been silently dropped
in the last rewrite. AsosAdjustmentWorker depends on the MRMS
precip cache, so both move together.
The single Era5MonthBatchWorker pinned an Oban slot for the full 30-45
min a CDS month-tile takes to assemble, and lost the work entirely on a
rolling deploy because the in-flight Task died with the pod. This splits
the flow into two tiny workers and persists the CDS job IDs so deploys
survive.
New pipeline:
1. Era5SubmitWorker (:era5_submit queue) — POSTs both CDS requests in
parallel, writes an `era5_cds_jobs` row with the returned job IDs,
and enqueues an Era5PollWorker scheduled +5 min. ~1s of real work.
Short-circuits when the month-tile is already cached or when a row
already exists (in-flight from a previous attempt).
2. Era5PollWorker (:era5_poll queue) — reads the row, calls
Era5Client.check_status/1 for both CDS job IDs, and:
- returns {:snooze, 300} if either job is still running (Oban
re-schedules without counting an attempt and releases the slot
immediately — a pod can keep dozens of tile-months in flight
without pinning workers)
- streams both GRIB files to disk via Req into: File.stream!,
decodes via Wgrib2.extract_grid_messages_from_file, bulk-inserts
via Era5BatchClient.decode_and_insert/6, deletes the row, and
DELETEs both completed jobs from CDS to free server-side quota
- if either leg CDS-reports failed, deletes the row + both CDS
jobs and returns {:error, reason}
Era5Client gains four testable building blocks:
submit_job/2 (bare POST → {:ok, job_id})
check_status/1 (GET → :running | {:done, src} | {:failed, reason})
download_source_to_file/3 (streams {:url, href} or writes {:body, bin})
delete_job/1 (DELETE /jobs/:id, treats 200/202/204/404 as :ok)
All Req calls now route through `era5_req_options` so tests can stub
CDS responses via Req.Test.stub(Era5Client, fn).
Era5MonthBatchWorker is retained as a thin forwarder to Era5SubmitWorker
so any jobs already in the :era5_batch queue on prod pods drain cleanly
on the next rolling deploy. Safe to delete in a follow-up.
Adds era5_cds_jobs table with a unique index on
(year, month, tile_lat, tile_lon) so duplicate submits collapse.
New queue config in runtime.exs:
era5_submit: local_limit 4, rate_limit 30/hour (burst protection)
era5_poll: local_limit 20 (polls are cheap GETs)
era5_batch: kept at 1 for legacy job drain, delete next cycle
Raise local_limit 2→4 (12 concurrent cluster-wide) and rate_limit 10→30
per hour. Each job makes 2 serial CDS requests, so active workers map 1:1
to CDS slots; 12 sits safely under CDS-Beta's ~16 per-user ceiling while
the rate gate stays above steady-state so it stops being the bottleneck.
The Pro Smart engine's global_limit tracker is stored inside
oban_producers.meta and is not reliably released when a pod exits
between job start and completion. After today's rollout two pods died
holding tracked slots for running era5_batch jobs; the survivors refused
to start any new ones because the cluster-wide tracker was stuck at
4/4 — with zero actual jobs executing — and the whole backfill froze
for 2+ hours.
Keeping rate_limit (10/hr) is enough to stay polite to Copernicus CDS.
local_limit stays at 2 per pod, so effective cluster concurrency just
scales with replica count instead of being capped by a global value
whose bookkeeping can't survive a restart.
IEM's RAOB endpoint stopped keeping Canadian stations current — most
stalled around Sep 2024 — which leaves a gap in refractivity/ducting
calibration over all the CW*/CY* stations already in weather_stations.
MSC's own tephi CSV is fixed-width and rendering-focused; the WMO TEMP
bulletins would need a full decoder. University of Wyoming serves the
same stations in a clean space-delimited format, is current, and
accepts the 3-letter station code (drop the leading C from Canadian
ICAO). Its output shape matches IemClient.parse_raob_json/1 exactly,
so it drops straight into Weather.upsert_sounding/2.
- New UwyoSoundingClient with URL builder, Req-based fetch_sounding,
and a fixed-width parse_sounding_html that extracts PRES/HGHT/TEMP/
DWPT/DRCT/SKNT from the <PRE> block and DateTime from the <H2> header.
- New CanadianSoundingFetchWorker: Oban batch worker that finds all
sounding stations whose code starts with C, picks the most recently
publishable 00Z or 12Z slot (90 min publish delay), calls UWYO per
station, and upserts with SoundingParams-derived refractivity/
ducting fields.
- Oban cron at 01:30Z and 13:30Z (dev + prod) to drive the worker.
- Req.Test plug wiring in config/test.exs.
- Captured Goose Bay fixture as test/support/fixtures/uwyo_sounding_yyr.html.
Also drops two implementation plans under docs/plans/:
- 2026-04-13-rdps-vertical-profiles.md: ~2 day plan for RDPS 10 km
Canadian ducting outside HRRR's Lambert footprint.
- 2026-04-13-hrdps-canadian-prop-grid.md: ~5 day plan for the full
HRDPS 2.5 km Canadian prop grid. Explicitly recommends doing RDPS
first.
TDD throughout, 19 new tests, 1318/1318 passing, credo strict clean.
- Tailwind source(none) wasn't scanning deps, so shadcn tokens
(border-border, bg-muted, text-foreground, ...) used by live_table
and sutra_ui weren't generated, leaving `border` to fall back to
currentcolor — a heavy dark border around every table in light mode.
Add @source directives for deps/live_table/lib and deps/sutra_ui/lib.
- Drop the hardcoded width:100% on the outer .select wrapper so the
per-page selector's Tailwind w-24 wins instead of stretching across
the header and overlapping the search box.
- Make whole rows clickable: global click delegator in app.ts forwards
tr clicks to the first <a href> in the row (the Actions "View" link),
skipping inner interactive elements. cursor + hover highlight added.
- Replace the default Prev/Next footer with a daisyUI .join/.btn-based
pager (MicrowavepropWeb.LiveTableFooter), wired up globally via
:live_table defaults so it applies to every live_table at once.
warm_grid_cache_and_broadcast was re-SELECTing 92k hrrr_profiles rows
with JSONB profile/duct_characteristics columns on every forecast hour.
Row decoding exceeded the 15s default DB connection timeout, killing
the worker before f01-f18 could run. Oban retried from f00 and got
stuck in a loop: for the last several hours, no forecast hours were
being written and even f00 was stale.
Build the weather cache rows directly from the in-memory grid_data
the worker already has (Weather.build_grid_cache_rows/2) and
GridCache.broadcast_put directly. No DB round trip, no JSONB decode.
Also widen the cron from hourly to every 3 hours so a full f00-f18
sweep (~95 min) can actually complete before the next run starts,
and drop the redundant prune_old_scores() at worker start
(PropagationPruneWorker already runs every 15 min). Add a 60s query
timeout to load_weather_grid_from_db as defense-in-depth for the
cold-cache fill path that still uses it.
Instead of shelling out to https://gridmap.org/locate/:callsign for
callsign → lat/lon lookups, ports the resolver pipeline from gridmap-web
into this project so the whole flow runs in-process.
New modules:
- Microwaveprop.Qrz — cache facade over the QRZ.com XML callsign API.
Looks up from qrz_callsigns first, falls back to a live fetch, and
upserts the result with a configurable cache_ttl_hours (default
168h / 7 days).
- Microwaveprop.Qrz.Client — HTTP/XML client against
https://xmldata.qrz.com/xml/current/. Holds the session key in an
Agent, transparently re-logs-in on :session_expired, and parses
responses via xmerl.
- Microwaveprop.Qrz.Callsign — Ecto schema for the qrz_callsigns
cache table, binary_id primary key per project convention.
- Microwaveprop.Qrz.Record — slim struct with only the 11 fields
we actually consume (identity, name, grid, address, lat/lon).
The full XML payload stays in the raw :data jsonb column for
anyone who wants the other ~40 QRZ fields.
- Microwaveprop.Geocoder — Req-based client against the Google Maps
Geocoding API. Only called as a fallback when QRZ has no explicit
<lat>/<lon> for the callsign.
- Microwaveprop.CallsignLocation — orchestrator. Reads the
callsign_locations cache, on miss calls Qrz then either uses QRZ's
coords directly or geocodes the formatted address, snaps to an
8-char Maidenhead grid via Microwaveprop.Radio.Maidenhead, and
upserts the result.
Microwaveprop.Radio.CallsignClient.locate/1 is rewritten to delegate
to CallsignLocation.lookup/1 and shape the response back to the
existing {:ok, %{callsign, gridsquare, lat, lon}} contract so the
callers in PathLive and RoverLive don't change.
Wiring:
- priv/repo/migrations/20260413000000_create_qrz_callsigns_and_callsign_locations.exs
creates qrz_callsigns and callsign_locations with unique indexes
on :callsign.
- Microwaveprop.Qrz.Client added to the application supervision tree
so the session Agent is started.
- :xmerl added to extra_applications so the release bundles it.
- config/test.exs wires Req.Test plugs for Microwaveprop.Qrz.Client
and Microwaveprop.Geocoder, forces cache_ttl_hours: 0 so the cache
never short-circuits test-level stubs, and supplies dummy QRZ
credentials.
- config/runtime.exs pulls QRZ_USERNAME / QRZ_PASSWORD / QRZ_AGENT
and GOOGLE_API_KEY from the environment so prod and dev can
configure both upstream keys out of band.
Tests (ported verbatim from gridmap-web):
- test/microwaveprop/qrz/callsign_test.exs — schema/changeset
- test/microwaveprop/qrz_test.exs — cache hit, cache miss, TTL
behavior, upsert on stale, error passthrough, case-insensitive
input
- test/microwaveprop/geocoder_test.exs — success, zero results,
request denied, transport error
- test/microwaveprop/callsign_location_test.exs — end-to-end flow
including the QRZ-lat/lon shortcut and the missing-address error
path
All 1294 tests still pass. Credo strict clean.
Wire up gurujada/live_table 0.4.1 as a dependency (alongside its
transitive sutra_ui and optional igniter requirement) with the
:live_table config pointing at our Repo and PubSub. Forced oban_web
override so the path-pinned vendored copy still wins over live_table's
hex constraint.
Convert the admin /users page to use LiveTable.LiveResource with
sortable, searchable columns for callsign, name, and email. Custom
renderers preserve the daisyUI admin badge and the pending/confirmed
date cell. Hidden id field stays in the query so the edit and delete
actions can target a record by id.
Updated the delete test to push the delete event directly with an id
payload; the stream dom_ids under live_table are random UUIDs so
tr#users-<id> selectors no longer work.
MRMS
----
Layer the NOAA MRMS PrecipRate product onto the score grid so rain fade
updates every 2 minutes instead of every hour alongside HRRR. New modules:
- Microwaveprop.Weather.MrmsClient: fetches the latest .grib2.gz off the
NCEP mirror (Req auto-decompresses so no gunzip step), writes the raw
GRIB2 to a temp file, and calls the existing wgrib2 wrapper with the
0.125 propagation grid spec to get interpolated cells. Returns a
%{{lat, lon} => mm_per_hour} map with missing-value sentinels dropped.
- Microwaveprop.Weather.MrmsCache: ETS-backed GenServer mirroring
ScoreCache/GridCache. Caches a single "current" entry keyed by
valid_time with PubSub broadcast so peer nodes stay in sync and only
the Oban leader pays the fetch + regrid cost.
- Microwaveprop.Workers.MrmsFetchWorker: cron every 2 minutes, short-
circuits when the cached valid_time already matches the newest file.
Microwaveprop.Propagation.AsosNudge.compute/4 now takes an optional
rain_grid. When a cell has MRMS rain >= 0.1 mm/hr it gets patched onto
the HRRR profile's `precip_mm` field (the scorer already reads it there)
and the cell is re-scored even with no ASOS station nearby. Cells with
MRMS rain below the threshold aren't touched so dry cells keep their
raw HRRR scores (which have the wind/sky/native-gradient signal that
isn't persisted on HrrrProfile rows and would otherwise be lost).
AsosAdjustmentWorker pulls MrmsCache on every tick and passes the grid
through to AsosNudge.compute/4. Also skips the IemClient error branch
that can never happen and handles the ASOS-empty + MRMS-empty case
explicitly. MrmsCache wired into the supervision tree; MrmsFetchWorker
cron entry added to config.exs and dev.exs.
Four new AsosNudge cases cover MRMS-only re-scoring, threshold gating,
and the wet/dry score delta.
Beacons 500
-----------
Beacon.format_freq/1 and format_mw/1 crashed on whole-number floats
(e.g. 24192.0) because `frac == 0.0` could become false under float
rounding while `trim_trailing_zeros/1` stripped the decimal point,
leaving a 1-element list that couldn't be destructured as [_, frac].
Shared format_number/1 helper handles integer input directly and
pattern-matches both the "int-only" and "int + frac" shapes.
Added stream_data property tests covering the whole microwave range for
both integers and floats to catch this class of bug before prod.
UTC clock flash
---------------
The /weather and /map UTC clocks were empty until the JS hook mounted
post-WebSocket, producing a several-second blank spot on initial load
and a clobber risk on sidebar re-renders. Mount now computes a
server-rendered `initial_utc_clock` string and the template seeds the
element with that plus `phx-update="ignore"` so LiveView morphdom won't
overwrite what the hook writes.
Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
- Forecast hour clicks showed only a small square of coverage because
:preload_forecast ran at mount using a hardcoded fallback bounding box
instead of the client's actual viewport. Move the preload trigger into
handle_event("map_bounds", ...) so the cache always matches the
viewport the client just asked about.
- Map overlay blinked on load/update because renderScores called Leaflet
GridLayer.redraw(), which removes every tile element before recreating
them. Repaint the existing tile canvases in place instead.
- Prod SMTP to mail.smtp2go.com was failing with an "unexpected_message"
TLS alert: gen_smtp wasn't sending SNI so the wildcard cert endpoint
rejected the handshake. Set tls: :always with explicit tls_options
(server_name_indication, versions, verify).
- Add contacts_inserted_at_desc_id_desc_idx so /contacts list runs as an
Index Scan (~0.4ms) instead of a Seq Scan + top-N heapsort (~77ms on
58k rows)
- Add Microwaveprop.Cache: tiny ETS-backed TTL cache for memoizing
expensive-but-stale-tolerant values
- Cache total_entries for unsearched /contacts page loads (30s TTL,
invalidated on insert)
- Cache the entire /contacts/map payload (pre-encoded JSON + band list),
moving load_contacts into Radio.contact_map_payload; invalidated on
new contact insert. Mount goes from ~150-300ms to ~5ms.
- Fix /map overlay blinking on load: reuse the Leaflet GridLayer across
renderScores calls and just redraw() against the updated ScoreGrid,
instead of removeLayer + new layer which flashed between tile regens
- Make mode optional on submission: schema change (null: true),
submission_changeset no longer requires it, blank strings normalise to
nil, CSV import accepts 6-column (no mode) or 7-column layouts
- core_components .input adds a red asterisk to labels when required,
giving users a visual cue for which fields must be filled on /submit
The ownership_timeout controls how long a process can hold a checkout,
but the per-query Postgrex timeout (default 15s) is separate. Oban
inline mode chains workers synchronously, so individual queries can
exceed 15s under concurrent test load.
- Rename all .js files to .ts in assets/js/
- Add interfaces for hook state, data structures, and event payloads
- Add type annotations to function parameters and return types
- Create type declarations for vendor deps (Leaflet, Chart.js, Phoenix, topbar)
- Update tsconfig.json for strict TypeScript checking
- Update esbuild entry point to app.ts
Release.backtest_all, climatology, native_derive now enqueue an
AdminTaskWorker job on the new :admin queue and return immediately.
Progress visible in Oban Web at /admin/oban.
Wire up the Oban Pro Smart engine in base + runtime config and add the
Pro migration (oban_producers, oban_crons, oban_queues tables + the
v1.6 partition/workflow index additions on oban_jobs).
On the Smart engine the era5_batch queue now has a global_limit of 4
in-flight jobs across the cluster and a rate_limit of 10 jobs per hour
so slow Copernicus CDS submit/poll cycles can't exhaust our daily
request budget when a backfill kicks off.
The slow CDS submit/poll/download cycle of the month-batch worker was
sharing the era5 queue with the cheap Era5FetchWorker router, and its
2 concurrent slots were permanently pinned by long-running batch jobs
while hundreds of router jobs starved. Give the batch worker its own
era5_batch queue (also 2 concurrent per pod) so the router has
dedicated capacity.
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
- Add on_the_air boolean to beacons (default true); surfaced as a
checkbox on the form, a badge on the index, and in the detail list.
- Render a Leaflet map on the beacon show page with a marker at the
beacon's lat/lon tooltipped with callsign + frequency. Marker is
green when on air, gray when off.
- Compute and draw a reception-range estimate as concentric signal-
strength rings. New Microwaveprop.Beacons.RangeEstimate solves a
link budget (FSPL + O2/H2O absorption from BandConfig) at five RX
thresholds (-100 to -145 dBm), then scales by 0.5 + score/100 from
the latest Propagation.point_detail at the beacon's grid square, so
current HRRR conditions shift the rings in or out.
- Re-enable the hourly PropagationGridWorker cron and freshness
monitor in dev.exs so dev actually has HRRR-backed scores to feed
the new estimator.
Generated Accounts context, User schema, and controllers via
phx.gen.auth. Adapted it for password-only login with required
email confirmation:
- Users have callsign (unique, uppercased), name, email, password
- Registration form fields: callsign, name, email, password, confirm
- Magic-link login path removed; login is email + password only
- After register, a confirmation email is sent and login is blocked
until the account is confirmed via the token URL
- Confirmation link logs the user in on first use
- SMTP2GO configured as the outgoing mailer in k8s prod
Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
Per-pod queues divided by 3 to maintain sane cluster totals.
Lifeline rescue reduced to 10 min to recover OOM orphans faster.
Memory limit 2Gi → 4Gi for HRRR wgrib2 processing.
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
- Unified queue_info helper shows "(N jobs in queue)" on all spinners
- Solar index auto-enqueues when missing on contact page
- Freshness monitor disabled in dev
- Fix test stubs for solar client
