Adds Microwaveprop.Propagation.AsosNudge: a pure IDW bias-field module
that takes ASOS observations + HRRR profiles and returns re-scored grid
rows for every cell within 250km of a reporting station. Upper-air
fields (min_refractivity_gradient, pwat_mm, hpbl_m, profile, duct
metadata) pass through unchanged so HRRR's signal isn't clobbered.
The old AsosAdjustmentWorker was unwired and buggy — nil'd out ~22% of
the scoring weight and wrote orphan timestamps. Replaced with a slim
worker that queries the latest HRRR valid_time, fetches live ASOS
currents, calls AsosNudge.compute/3, and upserts onto
(lat, lon, valid_time, band_mhz) so nudged values overwrite the HRRR
hour cleanly instead of polluting available_valid_times. After each
upsert it warms ScoreCache and broadcasts propagation:updated so live
/map clients refresh.
Cron hooked up every 10 minutes in config.exs and dev.exs. Also cleaned
up the stale "dev has propagation disabled" note in CLAUDE.md.
13 new AsosNudge unit tests cover: residual computation (co-located,
out-of-grid, nil fields), IDW weighting (single station, far station,
two equidistant stations, nil component handling), upper-air
preservation, and the compute/3 entry point's shape and radius filter.
Drive-by Styler formatting touched a handful of unrelated files from
`mix format`.
- Forecast hour clicks showed only a small square of coverage because
:preload_forecast ran at mount using a hardcoded fallback bounding box
instead of the client's actual viewport. Move the preload trigger into
handle_event("map_bounds", ...) so the cache always matches the
viewport the client just asked about.
- Map overlay blinked on load/update because renderScores called Leaflet
GridLayer.redraw(), which removes every tile element before recreating
them. Repaint the existing tile canvases in place instead.
- Prod SMTP to mail.smtp2go.com was failing with an "unexpected_message"
TLS alert: gen_smtp wasn't sending SNI so the wildcard cert endpoint
rejected the handshake. Set tls: :always with explicit tls_options
(server_name_indication, versions, verify).
Release.backtest_all, climatology, native_derive now enqueue an
AdminTaskWorker job on the new :admin queue and return immediately.
Progress visible in Oban Web at /admin/oban.
Wire up the Oban Pro Smart engine in base + runtime config and add the
Pro migration (oban_producers, oban_crons, oban_queues tables + the
v1.6 partition/workflow index additions on oban_jobs).
On the Smart engine the era5_batch queue now has a global_limit of 4
in-flight jobs across the cluster and a rate_limit of 10 jobs per hour
so slow Copernicus CDS submit/poll cycles can't exhaust our daily
request budget when a backfill kicks off.
The slow CDS submit/poll/download cycle of the month-batch worker was
sharing the era5 queue with the cheap Era5FetchWorker router, and its
2 concurrent slots were permanently pinned by long-running batch jobs
while hundreds of router jobs starved. Give the batch worker its own
era5_batch queue (also 2 concurrent per pod) so the router has
dedicated capacity.
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
Per-pod queues divided by 3 to maintain sane cluster totals.
Lifeline rescue reduced to 10 min to recover OOM orphans faster.
Memory limit 2Gi → 4Gi for HRRR wgrib2 processing.
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
ASOS re-scoring was upserting 458K records every 10 min while losing
the refractivity gradient (the key ducting indicator). HRRR hourly
provides all surface variables plus vertical profile data.
458K-record upsert held a connection for the entire transaction,
exceeding the 15s Postgrex timeout on prod. Set transaction timeout
to infinity and increase prod pool from 10 to 20.
- Add enqueue_for_qso/1 to directly enqueue weather/HRRR/terrain/IEMRE
jobs for a single user-submitted QSO (no cron, no bulk processing)
- Submit flow calls enqueue_for_qso instead of generic enqueue worker
- Add enrichment queues to prod config for on-demand processing
- Guard against HRRR fill values in store_hrrr_profiles (fixes badarith)
- Filter QSOs without pos2 in build_terrain_jobs
Only run live scoring (propagation, commercial, solar) in prod.
Backfill queues (weather, hrrr, iemre, terrain, enqueue) and the
QsoWeatherEnqueueWorker cron are dev-only for historical enrichment.
When a local .hgt tile is missing, download it from the public AWS S3
skadi bucket, decompress with zlib, and write to the tiles directory
before retrying the lookup. Falls back to Open-Meteo/OpenTopo APIs if
the download fails.
