diff --git a/CLAUDE.md b/CLAUDE.md index ce0d66e4..75320e14 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,9 +4,28 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Overview -Microwaveprop is a Phoenix 1.8 web application (Elixir ~> 1.15) with LiveView, PostgreSQL via Ecto, and Tailwind CSS v4 + daisyUI for styling. HTTP server is Bandit. +Microwaveprop is a Phoenix 1.8 web application for the North Texas Microwave Society (NTMS) that predicts microwave radio propagation conditions (10-241 GHz). It helps amateur radio operators plan and coordinate microwave contacts by showing real-time and forecast propagation maps. -**All UI must be built with Phoenix LiveView.** Write idiomatic Elixir throughout. Phoenix uses esbuild for JS bundling — never run npm or use Node.js tooling. +**Stack:** Elixir ~> 1.15, Phoenix 1.8 + LiveView, PostgreSQL via Ecto, Tailwind CSS v4 + daisyUI, Bandit HTTP server, Nx/Axon/EXLA for ML. esbuild for JS bundling — never npm. + +### What It Does + +1. **Propagation Map** (`/map`) — Full-screen Leaflet map with colored overlay showing propagation scores (0-100) per grid point across CONUS at 0.125° resolution. Includes 18-hour forecast timeline, band selector (10-241 GHz), Maidenhead grid overlay, point detail with factor breakdown and forecast sparkline, and terrain viewshed analysis. + +2. **QSO Submission** (`/submit`) — Users submit microwave contacts. On submission, the system automatically enqueues weather/HRRR/terrain/IEMRE enrichment jobs for that QSO. + +3. **Algorithm Documentation** (`/algo`) — Renders `algo.md` as HTML. Documents the scoring algorithm, meteorological foundations, ITU-R models, and calibration data. + +4. **Hourly Scoring Pipeline** — Every hour, the PropagationGridWorker fetches HRRR forecast hours f00-f18, computes propagation scores across all bands, and pushes updates to connected clients via PubSub. + +### Key Data Sources + +- **HRRR** (High-Resolution Rapid Refresh) — 3 km NWP model from NOAA AWS S3. Surface + pressure level profiles. Analysis + 18-hour forecasts. +- **ASOS** (Automated Surface Observing System) — Surface weather via Iowa Environmental Mesonet. +- **RAOB** (Radiosonde) — Upper-air soundings via IEM. +- **IEMRE** — Gridded hourly reanalysis at 0.125° resolution. +- **SRTM** — 90m terrain elevation for path analysis (local tile cache or API fallback). +- **Commercial Links** — 7 microwave links near DFW polled via SNMP at 5-min intervals. ## Commands @@ -22,7 +41,7 @@ iex -S mix phx.server # with IEx shell mix test # Run a single test file -mix test test/microwaveprop_web/controllers/page_controller_test.exs +mix test test/microwaveprop/terrain/terrain_analysis_test.exs # Run previously failed tests mix test --failed @@ -34,7 +53,6 @@ mix format mix credo # Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test) -# Runs in test env mix precommit # Database @@ -43,6 +61,9 @@ mix ecto.migrate mix ecto.gen.migration migration_name_using_underscores mix ecto.reset # drop + setup +# Run propagation grid manually +mix propagation_grid + # Assets mix assets.build mix assets.deploy # minified + digest @@ -50,21 +71,88 @@ mix assets.deploy # minified + digest ## Architecture -Standard Phoenix 1.8 structure: +### Directory Structure -- `lib/microwaveprop/` - Business logic (contexts, schemas, repo) - - `repo.ex` - Ecto PostgreSQL repository - - `application.ex` - OTP supervision tree -- `lib/microwaveprop_web/` - Web layer - - `router.ex` - Routes and pipelines. The `:browser` scope is aliased to `MicrowavepropWeb` - - `components/core_components.ex` - UI component library (flash, button, input, table, list, icon) - - `components/layouts.ex` - Layout components; `Layouts` is aliased in `microwaveprop_web.ex` - - `controllers/` - Traditional controllers -- `config/` - Environment configs (`dev.exs`, `test.exs`, `prod.exs`, `runtime.exs`) -- `assets/` - JS (esbuild, not npm) and CSS (Tailwind v4). Only `app.js` and `app.css` bundles are supported -- `test/support/` - `ConnCase` and `DataCase` test helpers +- `lib/microwaveprop/` — Business logic (contexts, schemas, workers) + - `propagation/` — Scoring engine: `scorer.ex`, `band_config.ex`, `grid.ex`, `model.ex` (ML), `freshness_monitor.ex` + - `weather/` — Data ingestion: `hrrr_client.ex`, `iem_client.ex`, `sounding_params.ex`, GRIB2 decoder + - `terrain/` — Path analysis: `terrain_analysis.ex` (ITU-R P.526-16), `viewshed.ex`, `elevation_client.ex` + - `radio/` — QSO schema, Maidenhead grid conversion + - `commercial/` — SNMP polling for commercial microwave links + - `workers/` — Oban background jobs for all data pipelines +- `lib/microwaveprop_web/` — Web layer + - `live/map_live.ex` — Main propagation map with forecast timeline + - `live/submit_live.ex` — QSO submission form + - `live/algo_live.ex` — Algorithm documentation page + - `live/qso_live/` — QSO list and detail views + - `plugs/remote_ip.ex` — X-Forwarded-For client IP extraction +- `config/` — Environment configs. `runtime.exs` has production Oban config (no backfill queues) +- `assets/js/propagation_map_hook.js` — Leaflet map, canvas heatmap, timeline, grid overlay +- `algo.md` — Full algorithm documentation with meteorological foundations +- `priv/models/` — Saved ML model weights (gitignored) -## Key Conventions (from AGENTS.md) +### Background Job Queues (Oban) + +| Queue | Workers | Purpose | +|---|---|---| +| `propagation` | PropagationGridWorker | Hourly HRRR fetch (f00-f18) + score computation | +| `weather` | WeatherFetchWorker | ASOS/RAOB data for QSO enrichment | +| `hrrr` | HrrrFetchWorker | HRRR profiles for individual QSOs | +| `terrain` | TerrainProfileWorker | ITU-R P.526-16 terrain diffraction analysis | +| `iemre` | IemreFetchWorker | IEMRE gridded weather for QSO enrichment | +| `commercial` | PollWorker | SNMP polling of commercial links | +| `solar` | SolarIndexWorker | Daily solar indices | +| `enqueue` | QsoWeatherEnqueueWorker | Batch enqueue enrichment jobs (dev cron only) | + +**Production** runs: propagation, commercial, solar, weather, hrrr, terrain, iemre queues. No cron backfill — enrichment is triggered by QSO submission only. + +**Dev** currently has propagation disabled for backfill. Re-enable by removing the Oban override in `config/dev.exs`. + +### Scoring Algorithm + +9-factor weighted composite score (0-100) per grid point per band: + +| Factor | Weight | Source | +|---|---|---| +| Humidity | 20% | HRRR surface temp + dewpoint | +| Time of day | 20% | UTC hour, cyclical | +| T-Td depression | 12% | HRRR surface | +| Refractivity | 10% | HRRR profile gradient (thresholds calibrated for HRRR resolution) | +| Sky cover | 10% | HRRR cloud cover | +| Season | 10% | Month, band-dependent (inverted for 24G+) | +| Rain | 8% | HRRR precipitation, ITU-R P.838-3 | +| Wind | 6% | HRRR 10m wind | +| Pressure | 4% | HRRR surface pressure | + +Humidity effect reverses by frequency: beneficial at 10 GHz (refractivity), harmful at 24+ GHz (absorption). + +### Terrain Analysis (ITU-R P.526-16) + +- Knife-edge diffraction loss (Eq. 31) +- Deygout 3-edge method for multiple obstacles +- Dynamic k-factor from HRRR refractivity gradient +- Profiles stored per QSO path (58k+ paths analyzed) + +### ML Model (Nx/Axon/EXLA) + +Skeleton feed-forward network for future propagation prediction: +- 13 features: 8 atmospheric + 4 cyclical temporal + 1 log-frequency +- Architecture: Dense(64) → Dropout(0.2) → Dense(32) → Dropout(0.1) → Dense(1, sigmoid) +- Weights saved to `priv/models/propagation_v1.nx` +- Not yet trained — scaffolding only + +### Data Flow + +``` +HRRR (NOAA S3) → PropagationGridWorker (hourly, f00-f18) + → store hrrr_profiles → compute scores → upsert propagation_scores + → PubSub broadcast → MapLive → JS hook → canvas heatmap + timeline + +User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers + → enrich QSO with atmospheric + terrain data +``` + +## Key Conventions ### Phoenix / LiveView - LiveView templates must start with `` wrapping all content @@ -88,7 +176,7 @@ Standard Phoenix 1.8 structure: - Assets are bundled via esbuild and Tailwind mix tasks — never use npm, npx, or Node.js directly - Tailwind v4: no `tailwind.config.js`, uses `@import "tailwindcss" source(none)` syntax in `app.css` - Never use `@apply` in CSS -- No inline `