LocationMap hook now adds a togglable grid layer (matching the contact
map pattern) so the rover-location detail page shows the standard
Maidenhead overlay alongside the marker.
Rover.update_location/3 and delete_location/2 now resolve any record
when the actor is_admin. The index + show LiveViews surface Edit/Delete
on every row for admin users, matching the existing owner UX.
Each row now has a View link (and the location cell links too) to a
new show LiveView that renders the full record alongside a Leaflet
map centered on the marker. Owners get a Delete button there as well.
Adds a Maidenhead grid input to the rover-location form. Typing a grid
fills in lat/lon (using square center); editing lat/lon recomputes the
grid at 6-char precision. The phx-change `_target` decides direction so
edits never clobber what the user just typed.
Without the deferral the cron-driven ETS flush GenServer fires every
7.5s alongside Prometheus scrapes, calls PromEx.get_metrics in a Task,
and dies with `Task.await ... time out` (hardcoded 10s) under contention.
Mirrors upstream PromEx.Plug, which calls defer_ets_flush after a
successful render specifically to suppress the cron while scraping is
healthy.
- Extract PathParser.valid_callsign?/1 and reuse from the calibration
Mix task instead of duplicating the regex.
- Document raise behavior on Aprs.recent_packets_with_paths/1 and
Aprs.station_positions/1 (already noted in moduledoc; @doc reinforces
for callers reading the function head).
- Note in the BandConfig guard comment why it sits before the Oban pause
(so a misconfig exits without leaving queues paused).
- Reword the band-mismatched-negatives note as a v1 limitation (no TODO
tag — codebase convention is zero TODOs).
- Print '<computed> / <attempted>' for both positive (HRRR-coverage) and
negative (random-baseline → HRRR) factor compute stages so an operator
can see how much of the corpus survived each filter.
- Guard at task entry: Mix.raise if BandConfig has no 144 MHz entry.
Without it the Recalibrator silently falls back to 10 GHz physics.
- random_baseline now draws from sample_size: max(n, 5_000) so small-n
fits don't cluster geographically.
- Wrap body in try/after Oban.resume_all_queues so iex -S mix workflows
don't leave Oban paused after the task returns.
Pulls verified RF hops from aprs.me's 24h packet retention, computes
factor vectors at each hop's midpoint via the existing Recalibrator,
and runs gradient descent against random-baseline negatives.
Dry-run: prints proposed weights but does NOT mutate band_config.ex.
Refuses to fit on <50 positive or negative samples.
- decode_packet_row no longer guards Decimal.to_float / DateTime.from_naive
with passthrough clauses — Postgrex returns Decimal for numeric and
NaiveDateTime for timestamp without time zone, so the WHERE clause
filters out the only path that could have hit the alternate branches.
- recent_packets_with_paths shifts :since to UTC before to_naive so a
caller passing a non-UTC %DateTime{} doesn't silently match the wrong
window.
- moduledoc now documents that the module raises on connection/SQL
errors and is unsafe in async contexts without a catch.
Q-construct now matches the explicit APRS-IS set (qAC/qAX/qAU/qAo/qAO/
qAS/qAr/qAR/qAZ/qAI) instead of any qA[A-Za-z]. WIDE/TRACE n-N now allows
n and N up to 7 per the APRS New-N paradigm (was [12]).
Three spec-compliance fixes to PathParser:
1. Q-construct regex now accepts lowercase letters (qAo, qAr, etc.).
The moduledoc lists qAo as a Q-construct to skip but the previous
~r/^qA[A-Z]$/ rejected it, leaving it to be misclassified.
2. self_loop?/2 now compares callsigns only; the spurious
src_pos == dst_pos clause would falsely suppress two distinct
stations that happen to share coarse coordinates (e.g. same building).
3. Strengthen the "lookup returned nil → drop hop, do NOT advance source"
test: extend the path to three tokens so the assertion actually proves
the third hop chains from the previous good source rather than from
the dropped one. The two-token version passed regardless of behavior.
Self-loop fixture updated to trigger via callsign equality (the only
remaining mechanism) instead of position equality.
Pure-Elixir parser that walks an APRS digipeater path left-to-right and
emits verified hops for callsigns carrying the used flag (`*`). Filters
Q-constructs, TCPIP/TCPXX, and routing aliases (WIDE/TRACE/RELAY/etc.);
handles anonymous-digi cases, missing station-position lookups, and
self-loops without advancing source past unattributable hops.
Used by upcoming 144 MHz calibration to attribute aprs.me receives to
real digipeater geometry without persisting any APRS data on our side.
- Empty-string env var (k8s secret resolves to '') was passing the nil
check and handing Postgrex an empty URL — now treated the same as unset.
- Drop runtime.exs warning; keep the single warning in application.ex
where the supervisor decision is made.
Wire up Microwaveprop.AprsRepo as an optional secondary Ecto repo that
points at the sister aprs.me Postgres for read-only access. This is the
foundation for upcoming 144 MHz calibration work that uses verified APRS
RF hops as ground truth.
* lib/microwaveprop/aprs_repo.ex declares a read_only repo; no schemas,
no migrations, no writes — Microwaveprop never persists APRS data.
* Application supervisor starts the repo only when configured (dev/test
via config blocks, prod via APRS_DATABASE_URL). A missing config logs
a warning and skips startup so the main app still boots.
* config/dev.exs + config/test.exs add local aprsme_dev / aprsme_test
endpoints. Test pool uses Ecto SQL Sandbox; tests that don't touch
APRS data don't need to check the connection out.
* config/runtime.exs guards the prod block on APRS_DATABASE_URL —
unset env logs a warning and leaves the repo unconfigured.
/map's PropagationMap hook pushes `map_bounds` on every moveend/zoomend.
When a user live-navigates from /map to /weather, an in-flight bounds
push can land on the new WeatherMapLive (the same LV channel survives
the navigation), and WeatherMapLive's `handle_event/3` had no clause
for it — the LV process crashed with FunctionClauseError, taking the
user's session down.
WeatherMapLive doesn't need viewport bounds — the /weather/cells HTTP
endpoint reads them off the URL the client builds — so the safe fix is
a no-op handler. Tested.
HRDPS at 0.5° step previously dropped its lat=50 row inside HRRR's lon
range, so its first painted cell sat at lat=50.5 (halfStep=0.25 →
50.25°N). HRRR's top row at lat=50 paints to 50.0625°N. The ~0.19°
(~21 km) blank strip between them was visible on /weather wherever the
viewport crossed the US-Canada border.
Make in_conus_bbox exclusive on the north edge (lat < LAT_MAX) so the
boundary row stays in HRDPS. The merge_prefer_hrrr layer dedupes by
exact (lat, lon) on the merged read path, and the canvas overlay paints
HRRR above HRDPS, so the new overlap is invisible everywhere HRRR has
data.
Won't take effect on prod until the next HRDPS chain run produces fresh
chunk files.
Past minute 30 of the wall clock the next forecast hour is closer to
utc_now() than the current hour, so the picker would land on (e.g.)
19:00Z when the user is at 18:43Z. HRRR has 19:00Z, but the HRDPS
chain only seeds the *current* hour (18:00Z), so the dual-source map
showed an empty Canadian half whenever we crossed the half-hour mark.
Switch to "most-recent valid_time <= now", with a fallback to the
closest future when nothing on disk is yet at-or-before now (cold
start). Hoist the picker into a public function so it can be unit-
tested deterministically — anchoring the integration test on
DateTime.utc_now() truncated to the hour would have only verified the
new behavior past minute 30.
/weather-ca delegates to the same picker so the two maps agree on
"current."
The /weather dual-source timeline picks times from HRRR's hourly
cadence, but HRDPS publishes 4×/day with multi-hour latency, so the
requested time almost never has an exact HRDPS dir on disk. The
controller was returning empty rows for HRDPS, leaving the Canadian
half of the map blank.
Snap to the closest HRDPS valid_time within a 6h window. /weather-ca
keeps exact-time semantics in practice because its timeline is built
from HRDPS-only listings, so the nearest is always the same time.
/weather now fetches both grids and paints each as its own canvas
overlay at its native cell size (HRRR 0.125°, HRDPS 0.5°). Server-side
chunk filtering means a US-only viewport gets an empty HRDPS pack and
vice versa — no wasted payload. /weather-ca's HRDPS-only mode is
unchanged.
Hook state moves from single (pack/overlay/abort) to per-source maps
keyed by "hrrr"|"hrdps". Render order is fixed (hrdps first, hrrr
second) so HRRR's higher-res canvas stacks on top along the border
where both sources cover the viewport.
The canvas overlay hard-coded a 0.0625° half-step when projecting each
cell to canvas pixels — correct for HRRR (0.125° native grid) but wrong
for HRDPS, which we downsample to 0.5° (0.25° half-step). At HRRR's
half-step every HRDPS cell got painted at 1/16th its actual area, so
the overlay over Canada looked like a sparse pinstripe of vertical
streaks instead of contiguous coverage.
Plumb halfStep through the WeatherCanvasOverlay constructor and pick
0.25 vs 0.0625 in the hook based on data-source.
The hourly :35 fire was reliably being missed in production: the prop
deployment is pinned to a single replica, and frequent ArgoCD deploys
restart the pod often enough that some fires land in the swap window.
Between 13:35Z and 17:25Z today the cron fired exactly zero times.
Switch to :05/:25/:45. Each fire calls seed_current_hour which is
idempotent on the (run_time, forecast_hour, kind, source) unique
index, so duplicate inserts in the same hour are no-ops at the DB
level and don't add cost on the happy path. Three shots per hour
gives two retry windows when a deploy kills the pod mid-cron-tick.
The 1-hour valid_time filter inherited from /weather is wrong for
HRDPS: the model publishes 4x/day with 3-4h publication latency and
the chain step adds ~13 min on top, so the freshest data on disk is
routinely 4-6h old. With the 1h cutoff in place, /weather-ca's
timeline was empty whenever there wasn't an active chain step running,
which left the JS hook with nothing to fetch. The sidebar already
shows the selected valid_time prominently — users can see for
themselves how stale the data is and don't need the page to hide it.
Recalibrator and FrontalAnalysis use Nx for tensor math, but Nx is
declared `only: [:dev, :test]` in mix.exs because neither module has a
prod entry point — Recalibrator is invoked by hand from IEx and
FrontalAnalysis is research scaffolding. The prod compile would emit
"Nx.to_number/1 is undefined" warnings on every Nx call. Add a
`@compile {:no_warn_undefined, Nx}` directive to both modules so the
prod build is clean without dragging Nx/Axon/EXLA (~hundreds of MB) into
the runtime image.
New LiveView at /weather-ca duplicates the /weather UI but defaults the
viewport to central Canada (55N, -100W, z=4) and renders the map div
with data-source="hrdps". The JS hook reads that attribute and appends
&source=hrdps to every cell fetch; the WeatherTileController routes
those reads through Weather.weather_grid_hrdps_at/2, which only reads
the <vt>.hrdps scalar dir and skips HRRR merging entirely. Timeline is
sourced from ScalarFile.list_valid_times_hrdps/0 so HRDPS-only hours
show up even when no HRRR file exists for the same valid_time.
Also drops "— Unified" from the algo.md H1.
The HRDPS pipeline as written was unusable in production: each chain
step took ~5 hours (not 30-90s as the dev-bench claim) because every
`wgrib2 -lon` batch redundantly JPEG2000-decodes ~150 records. With 57
batches × 18 forecast hours, a single cycle would take days to drain.
The /weather Canadian view only needs the current hour, so the cheapest
fix is to stop seeding the rest of the chain entirely:
* Rust grid: HRDPS_STEP=0.5 (was 0.125) drops point count 16× to ~3.5k.
Coarser cells than HRRR, but /weather gets visible Canadian coverage
inside one chain step (~20 min) instead of never.
* GridTaskEnqueuer.seed_current_hour/2 inserts exactly one forecast row
whose valid_time is closest to `now`. fh is clamped to 1..18 so the
Rust F00Reserved branch never fires.
* HrdpsGridWorker switches to seed_current_hour and drops the
6-hour-cycle cron in favor of HH:35 hourly fires. Reseeds are
idempotent on the 4-column unique key.
Also fixes a pre-existing credo "nested too deep" in
ScalarFile.read_point by extracting lookup_in_chunk + hit_or_false
helpers — happened to be on the read path that surfaces this work, so
folding it into the same commit.
The prior Docker build re-compiled all 546 transitive Rust deps from
scratch on every push because target/ is intentionally not cache-mounted
(Stream C disk-pressure failures). Two changes drop the cold-build cost
substantially:
* cargo-chef splits dep compilation into a separate Docker layer keyed
on recipe.json (derived from Cargo.toml/Cargo.lock). Source-only
pushes — the common case — now reuse the cooked deps as a normal
layer instead of re-running cargo for every crate.
* clippy now runs in --release profile so it shares artifacts with
`cargo test --release` and `cargo build --release`. Previously clippy
ran in dev profile, forcing a full second compile of every dep before
the release build could start.
Verified `cargo clippy --release --all-targets -- -D warnings` is clean
locally — no new lints surface under the release profile.
wgrib2 3.8.0's CMakeLists.txt:93 calls find_package(g2c 2.3.0), so the
previous 2.1.0 pin makes both Dockerfiles error out with "Could not find
a configuration file for package g2c that is compatible with requested
version 2.3.0". Bumping to 2.3.0 across both the Elixir and Rust images
since their wgrib2-builder stages stay in lockstep.
Latest-version sweep across the Oban stack. Final state:
oban 2.22.1 (hexpm)
oban_pro 1.7.0 (vendored, latest in licensed repo)
oban_met 1.1.0 (vendored, latest in licensed repo and hexpm)
oban_web 2.12.3 (vendored from public hexpm)
oban_web was re-vendored via `mix hex.package fetch oban_web 2.12.3
--unpack` and dropped into vendor/oban_web. No local mods to
preserve — vendor commit history shows the dir was a vanilla
import.
Aligning vendoring style with the sibling towerops-web repo. They
have the same vendor/oban_pro + oban_met + oban_web setup but no
`override: true` on oban_web — because they don't have live_table.
We do (live_table requires `oban_web ~> 2.11` from hexpm), so the
override is load-bearing here. Comment captures the reason so a
future deps cleanup doesn't assume it's vestigial and remove it.
Previous commit (3c988a5f) switched oban_pro to the licensed hex
repo at deploy time. Reverting that — keep all three Pro packages
vendored so prod image builds don't depend on oban.pro reachability
or auth on every CI run.
oban_pro 1.7.0 dropped into vendor/oban_pro via
`mix hex.package fetch oban_pro 1.7.0 --repo=oban --unpack`. mix.exs
goes back to `path: "vendor/oban_pro"` (matching oban_met / oban_web,
which were already vendored — and stay vendored since the licensed
repo only has older versions of those: 0.1.11 / 2.10.6 vs the
1.1.0 / 2.12.1 we vendor).
Schema migration from 3c988a5f stays — 1.7.0 tables/indexes are
already applied. No code changes.
Oban Pro 1.7.0 ships database-backed workflow tracking, replaces the
generated `uniq_key` / `partition_key` columns with expression
indexes, and adds new partial indexes for query performance.
Switches oban_pro from the vendor/ tree to the licensed `oban` hex
repo (auth already configured via `mix hex.repo`). vendor/oban_pro
deleted — the vendoring was a 1.6.x stopgap, 1.7 is the first
version we land directly from hex.
oban_met (1.1.0) and oban_web (2.12.1) stay vendored — they're
not published to the licensed `oban` repo (verified by 404 on
`mix hex.package fetch`).
Migration 20260430142341 runs `Oban.Pro.Migration.up(version: "1.7.0")`
which creates the workflow tables and the new indexes. Verified on
dev + test DBs. Production is well below the size threshold the v1.7
guide flags for split-migration handling, so the inline default
suffices.
No code changes for the breaking-change list — the codebase only
references Oban.Pro.Engines.Smart and Oban.Pro.Plugins.DynamicLifeline,
both unchanged in 1.7. We don't use Workflow.after_cancelled/2 or
Oban.Pro.Workers.Chunk.
Test suite stays at 3132/3132.
Adds lat/lon/zoom URL params to /weather so reload / share-link
restores the exact map state. The existing ?layer= param keeps
working alongside.
LiveView side:
- mount() reads lat/lon/zoom from query params, clamps to valid
ranges (lat -90..90, lon -180..180, zoom 4..10), and falls back
to DFW@z7 when missing or invalid. Initial values flow to the
hook via three new data attributes.
- handle_event("viewport_changed", ...) accepts {lat, lon, zoom}
pushed by the hook on debounced moveend, clamps + rounds, and
push_patches the URL with replace: true so browser history
doesn't grow an entry per pan tick.
- build_path/2 helper centralises URL construction so select_layer
and viewport_changed both produce a deterministic param order
(layer, lat, lon, zoom). Tests assert exact URLs.
JS hook side:
- mounted() reads data-initial-lat/lon/zoom from the el dataset
with parseFloat/parseInt + Number.isFinite gates, then seeds
Leaflet's center/zoom from those instead of the previous
hardcoded (32.897, -97.038, 7).
- moveend handler now also schedules a 400 ms-debounced
pushEvent("viewport_changed", {lat, lon, zoom}) so the URL
catches up with whatever the user panned/zoomed to. Separate
debounce timer from the cells fetch since the URL push and the
network fetch have different cadence sweet spots.
Tests cover: initial seeding from URL params, clamping of bad
values, viewport_changed event producing the right patched URL,
and the original layer-shareability test (now expecting the full
viewport in the URL).
Production wgrib2 3.8.0 still hit `*** FATAL ERROR: packing type 40
not supported ***` on HRDPS files. ldd confirmed wgrib2 was linking
only libc/libm — no OpenJPEG, no Jasper, no g2c.
wgrib2 3.8.0 cmake doesn't have USE_OPENJPEG or USE_JASPER options.
The legacy flag names we passed were silently ignored. The actual
knob is USE_G2CLIB_LOW, which links wgrib2 to the external g2c lib
(already built in the previous stage with OpenJPEG support). Dropped
the bogus USE_PNG / USE_OPENJPEG flags and added USE_G2CLIB_LOW=ON
in both Dockerfiles.
Verified by reading wgrib2 v3.8.0's CMakeLists.txt: the documented
JPEG2000 path is "USE_G2CLIB_LOW: Use g2c low-level decoders
(png,jpeg2000)?".
Reproduced in production pod 2026-04-30: wgrib2 3.6.0 crashes with
`free(): invalid size` when -lon point extraction is run against any
HRDPS rotated lat/lon GRIB2 file, regardless of batch size (failed at
N=200, N=1000, N=2000). Output before the abort is denormal f32 garbage
(val=4.93944e-41) — wgrib2's internal record buffer is being corrupted.
Same files extract cleanly with wgrib2 3.8.0 locally. Bumping the build
arg in both Dockerfile (Elixir image, used by hrrr-point-rs/per-QSO
HRDPS lookups) and rust/prop_grid_rs/Dockerfile (Rust grid worker) so
both consumers get the fix.
Also bumped POINT_BATCH 200 → 1000 in decoder.rs. With wgrib2 3.8.0
stable at 1000+ point batches, this drops the chain-step invocation
count from 286 → 57 per step. Comment captures the reproduction
context so a future bisection finds it quickly.
Production observation 2026-04-29: HRDPS chain steps using
decoder::extract_grid (wgrib2 -lola) ran at 1000m CPU for 10+ minutes
per task without producing any output. The -lola full-grid
interpolation against HRDPS's rotated lat/lon source grid is far
slower than against HRRR's Lambert source — wgrib2 appears to rebuild
the projection table per record.
The probe (lib/mix/tasks/hrdps_probe.ex, deleted in 1da78a80) had
already proved that wgrib2 -lon at five Canadian cities completes in
under a second. Add decoder::extract_points that uses the same -lon
flag for the HRDPS-only point set (~57k cells), batched at 2000 points
per wgrib2 invocation to stay under Linux ARG_MAX.
run_chain_step_hrdps now calls extract_points instead of extract_grid.
The post-extract HashSet filter to hrdps_only_points is gone —
extraction is already restricted to those cells. Per-cell wall time
should drop from "never completes" to ~30-90 s for 57k cells.
extract_grid stays unchanged — it works correctly for HRRR's CONUS
Lambert source.
Tests: parse_lon_output exercises the wgrib2 -s -lon output shape and
proves multi-record / multi-point parsing. parse_lon_segment unit
tests cover the undefined-value sentinel and the snap-back rejection
that ignores cells wgrib2 returned that don't correspond to anything
in the requested batch.
HPA: prop-grid-rs and hrrr-point-rs were sitting at 4 pods for
~16 min after the hourly chain spike — 10-min scaleDown stabilization
window plus a 1-pod-per-2-min drain policy. Both deployments stayed
visibly over-provisioned even when CPU dropped to 1m/pod.
Tighten scale-down to: 2-min stabilization (long enough that a real
multi-cycle surge doesn't immediately collapse) + Percent: 100 per
60s drain (back to min=1 within 1 minute of stabilization clearing).
Net: spike → scale up to handle queue (unchanged) → back to 1 pod
within ~3 min of spike ending. scaleUp left untouched.
CLAUDE.md updates so this doesn't bite again:
- Document `cargo clippy --all-targets -- -D warnings` as part of
the local Rust precommit ritual. The HRDPS commits in this branch
passed `cargo build` locally but failed CI on lints (manual_range_
contains, iter_kv_map). Adding the recipe under Commands plus a
Rust subsection in Key Conventions so future Rust touches don't
ship clippy regressions.
- Added a Testing note about not anchoring relative-time fixtures on
utc_now-truncated-to-hour — the weather_map_live one-hour-cutoff
test (fix in 2768fc68) was a textbook example: 50% flake rate
driven entirely by which minute of the hour the test ran in.
CI runs cargo clippy with -D warnings. The yesterday's HRDPS commits
introduced two pedantic-but-blocked lints:
- grid.rs:97 in_conus_bbox used `>=`/`<=` chain — clippy's
manual_range_contains says use (LAT_MIN..=LAT_MAX).contains(&lat).
- pipeline.rs:426 backfill_dpt_from_depr iterated `cell.iter()` and
threw the values away — clippy's iter_kv_map says use
cell.keys().filter_map(...). Tightened the inline format strings
to `{depr_key}` instead of the deprecated `{}`-with-arg form to
avoid a follow-up useless_format complaint on the same touch.
cargo clippy --all-targets -- -D warnings passes; cargo test
--release stays at 158 passed.
The /weather map now shows HRRR-derived data wherever HRRR has
coverage and falls back to HRDPS-derived data for Canadian cells
outside HRRR's bbox. North of the 49°N HRRR-coverage line the map
stays populated instead of going blank.
Rust:
- weather_scalar_file::dir_for_hrdps + write_atomic_hrdps land HRDPS
scalar chunks at `<vt>.hrdps/` so HRRR's `<vt>/` write isn't
clobbered. The HRRR writer wipes its dir before each write — both
sources need their own dir to coexist.
- pipeline::run_chain_step_hrdps now derives a ScalarRow for each
scored cell (same derive_row that HRRR uses; same wire format) and
writes them via write_atomic_hrdps alongside the .hrdps.prop score
files.
Elixir:
- ScalarFile.dir_for_hrdps + read_bounds + read_point + exists? all
walk both `<vt>/` (HRRR) and `<vt>.hrdps/` (HRDPS). read_bounds
concatenates with HRRR-precedence on overlap (defensive — Rust
pipeline doesn't write HRRR-overlap cells, but cheap to enforce).
- Weather.weather_grid_at unchanged: it routes through ScalarFile,
which now transparently returns the union. GridCache stays
per-valid_time so a single cache entry holds both regions.
Tests cover four scenarios: HRRR-only, HRDPS-only, both-present
union, and HRRR-precedence on the same cell.
Stages 6, 7, 10 of the HRDPS plan, plus the Elixir read-side merge
that makes Rust's `.hrdps.prop` writes show up on /map.
Read side:
- `ScoresFile.path_for_hrdps/2` + `read_hrdps/2` — companions to the
HRRR equivalents. Same decode path; different filename.
- `ScoresFile.read_bounds/3` now reads HRRR + HRDPS files and
concatenates the cells. Files are disjoint by construction
(Grid.hrdps_only_points excludes CONUS) so no de-dup needed.
- `ScoresFile.read_point/4` checks `.prop`, then `.hrdps.prop`, then
legacy `.ntms` — first hit wins. Cells outside their owning region
read as no-data in the other file's grid, so the order doesn't
matter for correctness.
- `Propagation.warm_cache_and_broadcast/2` reads both files and
merges before broadcasting to the cluster ScoreCache. A single
missing file (HRDPS pre-cycle, HRRR briefly absent) is now OK —
the cache warms with whichever side is available.
Activation:
- runtime.exs cron: HRDPS seed worker fires at HH:35 of each cycle
hour (05:35Z, 11:35Z, 17:35Z, 23:35Z) — 5h after cycle, 30 min
staggered from HRRR's */15 schedule.
- HrdpsGridWorker moduledoc updated: no longer dormant; Rust
prop-grid-rs handles source='hrdps' rows.
- map_live North America bbox extended to (24.5, 60.0, -141.0, -52.0)
for the out-of-coverage banner check; visitors anywhere in CONUS
or the Canadian extent now stay banner-free.
Cleanup:
- mix hrdps.probe deleted — its risks are retired and the production
HrdpsClient covers the same surface.
What's still in motion (not blocking the Canadian /map):
- Per-valid_time profile + scalar artifacts for HRDPS (would let
/weather show Canadian temperature/refractivity layers). Rust
pipeline currently skips those so the HRRR-written companion files
aren't clobbered — separate stage when /weather UX is decided.
- FreshnessMonitor HRDPS staleness tracking — HRDPS has its own cron
and a different cadence than HRRR; HRRR-stale logic doesn't apply
cleanly. Defer until prod tells us what alarms we actually want.
Stages 4-5 of the HRDPS plan. Lands the Rust forecast-hour pipeline
for source='hrdps' grid_tasks rows.
What ships:
- `hrdps_fetcher.rs` — sibling of fetcher.rs for ECCC's MSC Datamart
shape: date-prefixed URLs, one variable per GRIB2 file, concurrent
fetch + byte-concat into a single multi-record blob. wgrib2 reads
multi-record files natively so the downstream decoder path is
unchanged. Mirrors lib/microwaveprop/weather/hrdps_client.ex.
- `grid::hrdps_only_points()` + `grid::hrdps_grid_spec()` — Canadian
cells (49-60°N, -141 to -52°W) minus the HRRR overlap. Disjoint
from conus_points by construction.
- `db::TaskSource` enum + `source` column on ClaimedTask. claim_next
surfaces source for forecast lane dispatch; claim_next_analysis
filters to source='hrrr' only (HRDPS analysis path is not yet
ported — would need native-duct + NEXRAD + commercial merges that
Rust doesn't have for HRDPS).
- `pipeline::run_chain_step_hrdps` — fetch combined blob, decode with
HRDPS grid spec, drop HRRR-overlap cells via a HashSet of
hrdps_only_points, back-fill DPT from DEPR (HRDPS publishes DEPR
as primary), score every band, write `<band>/<iso>.hrdps.prop`.
Skips per-valid_time profile + scalar artifacts since those would
clobber HRRR's at the same valid_time — surfacing HRDPS on /weather
is a separate stage.
- worker.rs dispatches forecast lane on (kind, source) — Hrrr →
run_chain_step, Hrdps → run_chain_step_hrdps. Analysis stays
HRRR-only. Worker constructs an HrdpsClient alongside HrrrClient.
- `scores_file::write_atomic_hrdps` writes to `.hrdps.prop` with the
Canadian grid spec via the new encode_with_spec. Coexists with the
HRRR `.prop` file for the same (band, valid_time).
163 unit tests pass; backfill_dpt_from_depr handles surface and
pressure-level levels with no double-fill when DPT exists.
CLAUDE.md adds HRDPS to Key Data Sources and the Background Job
Queues table with explicit "dormant until Rust ships" framing.
The 2026-04-29 plan gets a Status snapshot at the top: which stages
shipped (0-3 Elixir + 8) vs which are deferred to the Rust port
sessions (4-7, 9). Activation path is one-paragraph: land Rust HRDPS,
add cron entry, widen map bounds, extend FreshnessMonitor.
Stage 3 (Elixir foundation) of the HRDPS plan. Lands the scaffolding
that lets the Rust prop-grid-rs HRDPS branch be wired up cleanly when
it ships, without further DB or Elixir-side changes.
What ships:
- `grid_tasks.source` column (default 'hrrr', backfill is a no-op).
Replaces the (run_time, forecast_hour, kind) unique index with a
4-column key (..., source) so HRRR and HRDPS analysis/forecast rows
for the same cycle coexist.
- `Grid.hrdps_only_points/0` — Canadian cells inside HRDPS bbox
(49-60°N, -141 to -52°W) but outside HRRR's CONUS bbox. Disjoint
from `conus_points/0` by construction so the two grids never
double-write the same (lat, lon).
- `Grid.point_source/1` — `:hrrr | :hrdps | :neither` classification
for a (lat, lon) pair. Routes per-QSO enrichment to the right NWP
source.
- `GridTaskEnqueuer.seed_with_analysis/2` and `seed/2` accept a
`:source` option; defaults to "hrrr" so existing callers are
unchanged.
- `HrdpsGridWorker` cycle seeder mirroring PropagationGridWorker's
shape but for HRDPS's 4×/day cadence. `:hrdps` queue added to base
Oban config.
What's deferred:
- The Rust `prop-grid-rs` worker doesn't yet branch on
`task.source`. Until that ships, `HrdpsGridWorker` stays out of
runtime.exs cron — it would just produce rows that the Rust worker
mishandles as HRRR. Module doc explicitly notes the dormant state.
- Map overlay extension and FreshnessMonitor HRDPS staleness deferred
to follow-up sessions once real data flows.
Activation path documented in the updated plan file.
Coverage cap recorded as Stage 8 decision: 60°N for v1 (SRTM stops
there). Arctic CDEM deferred to a follow-up plan.
Stage 2 of the HRDPS plan. Mirror of hrrr_profiles for HRDPS-derived
rows so HRDPS can be dropped/reprocessed without disturbing the 4500+
HRRR profiles already in prod. Same column set, same indexes,
PARTITION BY RANGE (valid_time) with quarterly partitions starting at
the deploy quarter.
Schema follows the @primary_key {:id, :binary_id, autogenerate: true}
convention and the same set of derived fields HrrrProfile carries
(surface_refractivity, min_refractivity_gradient, ducting_detected,
duct_characteristics, etc.) so the rest of the pipeline can consume
HRDPS-derived rows interchangeably with HRRR-derived rows.
Unique constraint on (lat, lon, valid_time) is enforced at the DB
level via the partitioned index; schema-level unique_constraint/2
matching against changeset errors doesn't apply because the
partition's index name varies per partition. Production upserts go
through `on_conflict: {:replace_all_except, ...}` per project
convention.
The test computed recent_t = (truncated_to_top_of_hour now) - 30min,
which is 30-90 min before real utc_now() depending on what minute the
test runs at. The LiveView's `cutoff = utc_now() - 1h` then dropped
recent_t whenever the real-time minute exceeded ~30, so the test
failed half the time.
Anchor recent_t at the top of the current hour — that's always within
the 1-hour window regardless of minute. The test still exercises the
"keep within last hour" path; just stops being clock-flaky.
Stage 1 of the HRDPS plan. Mirrors HrrrClient.fetch_grid/3's signature
so PropagationGridWorker can call either polymorphically.
Architectural shape per the probe wedge findings:
- ECCC's date-prefixed datamart URL structure
(dd.weather.gc.ca/{YYYYMMDD}/WXO-DD/model_hrdps/...)
- One-variable-per-file model — fetch ~14 small GRIB2s concurrently,
byte-concatenate into a single multi-record binary, hand to
Wgrib2.extract_points_from_file. wgrib2 reads multi-record files
natively so no -merge step is needed.
- wgrib2's -lon flag handles the rotated-lat/lon reprojection
internally — no manual rotation math.
Variable catalog covers the surface set the propagation scorer reads
plus the 1000-700 mb pressure-level set used by the refractivity
gradient. PWAT is unavailable in HRDPS f000 inventory; the scorer's
PWAT factor will fall back to its default for HRDPS-derived points.
DEPR (T-Td depression) is treated as an alternate primary; when DPT
is absent build_profile_from_extracted derives dewpoint as
temp - depression so downstream consumers see the same shape as HRRR.
Plan task 1.6 (CYYR projection sanity check vs UWYO sounding) is
covered by the live probe at lib/mix/tasks/hrdps_probe.ex which
already validated wgrib2's rotated-grid handling against five
Canadian cities; the production implementation routes through the
same Wgrib2.extract_points_from_file path that the probe used.
