graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
372 commits 1 branch 0 tags 8.1 MiB
Commit graph

372 commits

This branch
This branch
All branches
Author SHA1 Message Date
Graham McIntire
430a07240e Fix ERA5 CDS API endpoint for new v1 API 
- Submit: /api/retrieve/v1/processes/{dataset}/execution
- Request body wrapped in {"inputs": ...}
- Poll: /api/retrieve/v1/jobs/{jobId}
- Status field: "status" not "state", values: accepted/running/successful/failed
- Download: GET {job_url}/results, follow asset href link
- Add ERA5 credentials to k8s secret manifest
 2026-04-07 12:32:24 -05:00
FluxCD
29e3c118e4 chore: update prop image to git.mcintire.me/graham/prop:main-1775582667-e47c6ac [skip ci] 2026-04-07 12:32:24 -05:00
Graham McIntire
bbf3e67d85 Add ERA5 to backfill dashboard and enrichment pipeline 
- ERA5 checkbox on /backfill page alongside HRRR/weather/terrain/IEMRE
- BackfillEnqueueWorker handles era5 type (targets hrrr_status=unavailable)
- ContactWeatherEnqueueWorker.build_era5_jobs/1 creates ERA5 fetch jobs
  for path points without existing ERA5 profiles
- Progress bar shows ERA5 candidate count (HRRR-unavailable contacts)
- Status panel shows ERA5 candidates and fetched profile count
 2026-04-07 12:24:21 -05:00
Graham McIntire
77397a1625 Add mix era5_backfill task and gitignore .envrc 
Enqueues ERA5 fetch jobs for contacts where HRRR is unavailable.
Deduplicates by rounded grid point and hour, skips existing profiles.

Usage: mix era5_backfill [limit]
 2026-04-07 12:24:21 -05:00
FluxCD
05e3702b1b chore: update prop image to git.mcintire.me/graham/prop:main-1775581463-2286725 [skip ci] 2026-04-07 12:24:21 -05:00
Graham McIntire
29fef2afed Add ERA5 reanalysis and RTMA data sources 
ERA5 (Copernicus CDS API):
- Era5Profile schema matching HRRR profile structure for interop
- Era5Client with async job submission, polling, GRIB2 download
- Era5FetchWorker (Oban queue: era5, max_attempts: 5)
- Unified lookup: Weather.best_profile_for_contact/1 tries HRRR
  first, falls back to ERA5 for pre-2014 contacts

RTMA (NOAA S3, 2.5km/15-min):
- RtmaObservation schema for surface-only fields
- RtmaClient with byte-range GRIB2 requests (same pattern as HRRR)
- RtmaFetchWorker (Oban queue: rtma, max_attempts: 10)
- Weather.find_nearest_rtma/3 for surface condition lookup

Both queues added to production Oban config (2 workers each).
 2026-04-07 12:04:19 -05:00
Graham McIntire
0623c339ca Path-integrated HRRR scoring across pos1/mid/pos2 
Contact scoring now uses all HRRR profiles along the path instead of
just pos1. Aggregation strategy:
- Best along path for beneficial factors (refractivity, pressure)
- Worst along path for harmful factors (rain, wind)
- Average for neutral factors (temp, dewpoint, PWAT, BL depth)

Scorer.path_integrated_conditions/2 merges multiple profiles into a
single conditions map. Falls back gracefully to single-profile scoring
when only one profile is available.
 2026-04-07 12:04:19 -05:00
FluxCD
b5d86ab58b chore: update prop image to git.mcintire.me/graham/prop:main-1775580351-12b8bdb [skip ci] 2026-04-07 12:04:19 -05:00
Graham McIntire
62d15f39d6 Update seasonal scores from sounding data, document commercial link patterns 
Soundings (10,935): October has strongest gradients (-350 N/km) and
65.7% ducting, raised seasonal score 82→88. February ducting (20.2%)
exceeds January (14.8%), raised 32→40.

Commercial links (12,068 samples): 68 GHz shows 3.9 dB diurnal swing,
11 GHz shows 1.7 dB with inverted multipath pattern, 24 GHz remarkably
stable at 0.9 dB. Diurnal sensitivity is non-monotonic with frequency.

RAOB gradients avg -265 N/km vs HRRR -107 (2.5x stronger) — confirms
HRRR misses thin surface ducts that soundings resolve.
 2026-04-07 11:45:46 -05:00
FluxCD
adb610686f chore: update prop image to git.mcintire.me/graham/prop:main-1775578557-be70d39 [skip ci] 2026-04-07 11:19:43 -05:00
Graham McIntire
b2f3098083 Data-driven algorithm refinements from full dataset analysis 
Analyzed 37,925 HRRR-matched contacts from 58,367 total.

- Remove shallow BL bonus (score 82 for HPBL < 300m): data shows
  medium BL (1000-2000m) produces longest contacts (222 km avg),
  not shallow (210 km). Refractivity fallback now uses default score.

- Refine pressure scoring bins: add <980 mb tier (score 88), steeper
  gradient from low to high. Contacts at <970 mb avg 242.7 km vs
  184.1 km at 990-1000 mb.

- Update algo.md calibration stats (58,367 contacts, 41M HRRR
  profiles, 58,361 terrain profiles) and document dataset bias
  (99.5% Aug-Sep).

- Create updates.md with full binned analysis tables for all factors.
 2026-04-07 11:19:43 -05:00
Graham McIntire
03aa72ae80 Fix livebook path resolution in bin/notebook 
Falls back to mix escripts directory when livebook isn't in PATH.
 2026-04-07 11:19:42 -05:00
Graham McIntire
3b7711f01d Auto-deploy analysis notebook as Livebook app 
Add app_settings metadata to notebook and set LIVEBOOK_APPS_PATH
so it shows up under Apps on the Livebook home page.
 2026-04-07 11:19:42 -05:00
Graham McIntire
cf9c0a5e04 Add bin/notebook script to start Phoenix + Livebook together 
Starts Phoenix with a named node and Livebook attached to it on
port 8081. Single command, Ctrl-C stops both.
 2026-04-07 11:19:42 -05:00
Graham McIntire
ab811ae463 Add mix notebook task for Livebook attached to dev node 
Run `mix notebook` for instructions to start Livebook at localhost:8081
connected to the running Phoenix node with full Repo/Scorer access.
 2026-04-07 11:19:42 -05:00
Graham McIntire
b204b08751 Rewrite analysis notebook to attach to running node 
Uses Repo, Scorer, BandConfig, Weather directly instead of raw SQL.
Scores all historical contacts with the real composite_score/2 and
includes factor heatmaps and weight sensitivity analysis.
 2026-04-07 11:19:42 -05:00
Graham McIntire
af5a48b797 Add Livebook notebook for propagation algorithm analysis 
Connects to local dev DB to analyze contacts, HRRR conditions,
terrain impact, commercial link correlation, and score sensitivity.
Ready for use once prod data is imported locally.
 2026-04-07 11:19:42 -05:00
FluxCD
6b9ef24c81 chore: update prop image to git.mcintire.me/graham/prop:main-1775566243-7f457e4 [skip ci] 2026-04-07 11:19:42 -05:00
Graham McIntire
214b706ba8 Remove icons from navigation links in map control panels 2026-04-07 07:50:39 -05:00
Graham McIntire
d1e42dd8a0 Smooth viewshed boundary to remove SRTM elevation spikes 
Apply 5-point circular moving average to reach_km values before
rendering the boundary polygon. Eliminates sharp spikes caused by
isolated terrain anomalies in SRTM data.
 2026-04-07 07:42:35 -05:00
FluxCD
114f25a461 chore: update prop image to git.mcintire.me/graham/prop:main-1775565490-61e4196 [skip ci] 2026-04-07 07:42:35 -05:00
FluxCD
f17dfc7270 chore: update prop image to git.mcintire.me/graham/prop:main-1775565429-f0340a2 [skip ci] 2026-04-07 07:42:35 -05:00
Graham McIntire
970bc29afe Remove stale propagation scores maintenance banner 2026-04-07 07:38:12 -05:00
Graham McIntire
51c73a3cbd Add periodic enrichment backfill cron and terrain reconciliation 
Runs BackfillEnqueueWorker every 30 minutes to pick up contacts with
pending/queued/failed enrichment status and enqueue missing jobs.
Before enqueueing, reconciles terrain contacts stuck in "queued" by
checking the terrain_profiles FK directly.
 2026-04-07 07:37:08 -05:00
FluxCD
ec82f4ae71 chore: update prop image to git.mcintire.me/graham/prop:main-1775513079-b5a87e0 [skip ci] 2026-04-07 07:37:08 -05:00
Graham McIntire
5bd2516c15 Bump terrain queue concurrency to 3 per pod 2026-04-06 17:04:38 -05:00
FluxCD
5583fb6167 chore: update prop image to git.mcintire.me/graham/prop:main-1775512945-6ab4ba7 [skip ci] 2026-04-06 17:04:37 -05:00
Graham McIntire
c68e1ba108 Re-enable hourly propagation grid scoring in production 2026-04-06 17:02:12 -05:00
FluxCD
da4979751d chore: update prop image to git.mcintire.me/graham/prop:main-1775512477-be9b890 [skip ci] 2026-04-06 17:02:12 -05:00
FluxCD
25df2e6766 chore: update prop image to git.mcintire.me/graham/prop:main-1775509847-dcf5190 [skip ci] 2026-04-06 17:02:12 -05:00
Graham McIntire
25d7ec317a Mark terrain status complete when profile already exists 
TerrainProfileWorker early-exited with :ok when a profile existed
but never updated the contact status, leaving contacts stuck at queued.
 2026-04-06 16:54:23 -05:00
FluxCD
00bacae2c2 chore: update prop image to git.mcintire.me/graham/prop:main-1775499376-15ec8af [skip ci] 2026-04-06 16:54:23 -05:00
Graham McIntire
1f7e9e18a4 Store stub records on empty weather fetches to prevent infinite backfill retries 
When ASOS/RAOB/IEMRE APIs return empty data, store a stub record so
dedup checks (has_surface_observations?, has_sounding?, has_iemre_observation?)
see coverage and don't recreate the same jobs on future backfill runs.
 2026-04-06 16:10:28 -05:00
Graham McIntire
c3819112fe Persist checkbox and limit state across backfill enqueue submissions 2026-04-06 12:46:08 -05:00
FluxCD
ef44109351 chore: update prop image to git.mcintire.me/graham/prop:main-1775494893-2fd0175 [skip ci] 2026-04-06 17:02:37 +00:00
Graham McIntire
98289ec16e Fix format_number reversing digits 
Enum.reverse/1 was reversing each chunk's characters, scrambling the
displayed numbers (e.g. 58,363 displayed as 85,363).
 2026-04-06 12:01:25 -05:00
FluxCD
f9e2ba294d chore: update prop image to git.mcintire.me/graham/prop:main-1775494509-975e6c9 [skip ci] 2026-04-06 16:56:34 +00:00
Graham McIntire
9ffeba3fc9 Suppress health check request logs 
/health gets hit constantly by k8s probes, drowning out useful logs.
 2026-04-06 11:55:01 -05:00
FluxCD
88783765f1 chore: update prop image to git.mcintire.me/graham/prop:main-1775494141-8768175 [skip ci] 2026-04-06 16:50:32 +00:00
Graham McIntire
4ba7269e57 Prioritize pending contacts over queued in backfill 
Queued contacts just need reconciliation (check if data exists),
while pending contacts need actual HRRR fetches. Process pending
first so real work happens before reconciliation.
 2026-04-06 11:48:55 -05:00
FluxCD
89eafcb1b0 chore: update prop image to git.mcintire.me/graham/prop:main-1775493753-3737842 [skip ci] 2026-04-06 16:43:31 +00:00
Graham McIntire
e99e585b66 Add type filter to backfill enqueue 
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
 2026-04-06 11:42:26 -05:00
FluxCD
ca6e836bd9 chore: update prop image to git.mcintire.me/graham/prop:main-1775491184-a80a981 [skip ci] 2026-04-06 16:00:21 +00:00
Graham McIntire
89ade2e8cc Bump HRRR queue concurrency from 1 to 5 per pod 
15 concurrent HRRR fetches across 3 pods to process the 16,870
contacts that need fresh HRRR data from NOAA.
 2026-04-06 10:54:45 -05:00
FluxCD
023fef4d8c chore: update prop image to git.mcintire.me/graham/prop:main-1775490031-fda468c [skip ci] 2026-04-06 15:41:17 +00:00
Graham McIntire
0e885001bc Widen HRRR profile spatial matching from 0.05° to 0.07° 
The propagation grid stores profiles at 0.125° intervals, so contacts
up to 0.0625° from a grid point would miss with the old 0.05° tolerance.
This caused has_hrrr_profile? and find_nearest_hrrr to fail matching
against existing data, leaving contacts incorrectly stuck as pending.
 2026-04-06 10:40:25 -05:00
FluxCD
a8f6a3b0e8 chore: update prop image to git.mcintire.me/graham/prop:main-1775489105-fbfd9ea [skip ci] 2026-04-06 15:26:12 +00:00
Graham McIntire
eeb92bcc19 Fix HRRR enrichment status stuck at queued/pending 
enqueue_for_contact unconditionally marked contacts as :queued even
when no HRRR jobs were needed (profiles already exist). Workers never
updated contact status to :complete after storing profiles. Contacts
got stuck and were never re-processed.

- Mark :complete when no jobs generated (data already exists)
- Include :queued in backfill enrichable states for reconciliation
 2026-04-06 10:24:56 -05:00
FluxCD
c9d1abafda chore: update prop image to git.mcintire.me/graham/prop:main-1775488547-a24e710 [skip ci] 2026-04-06 15:16:09 +00:00
Graham McIntire
c98aabfc15 Add BackfillEnqueueWorker tests (0→100% coverage) 2026-04-06 10:15:38 -05:00