prop/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Microwaveprop is a Phoenix 1.8 web application for the North Texas Microwave Society (NTMS) that predicts microwave radio propagation conditions (10-241 GHz). It helps amateur radio operators plan and coordinate microwave contacts by showing real-time and forecast propagation maps.

Stack: Elixir ~> 1.15, Phoenix 1.8 + LiveView, PostgreSQL via Ecto, Tailwind CSS v4 + daisyUI, Bandit HTTP server, Nx/Axon/EXLA for ML. esbuild for JS bundling — never npm.

What It Does

1. Propagation Map (/map) — Full-screen Leaflet map with colored overlay showing propagation scores (0-100) per grid point across CONUS at 0.125° resolution. Includes 18-hour forecast timeline, band selector (10-241 GHz), Maidenhead grid overlay, point detail with factor breakdown and forecast sparkline, and terrain viewshed analysis.

2. QSO Submission (/submit) — Users submit microwave contacts. On submission, the system automatically enqueues weather/HRRR/terrain/IEMRE enrichment jobs for that QSO.

3. Algorithm Documentation (/algo) — Renders algo.md as HTML. Documents the scoring algorithm, meteorological foundations, ITU-R models, and calibration data.

4. Hourly Scoring Pipeline — Every hour, the PropagationGridWorker fetches HRRR forecast hours f00-f18, computes propagation scores across all bands, and pushes updates to connected clients via PubSub.

Key Data Sources

• HRRR (High-Resolution Rapid Refresh) — 3 km NWP model from NOAA AWS S3. Surface + pressure level profiles. Analysis + 18-hour forecasts.
• HRDPS (High Resolution Deterministic Prediction System) — 2.5 km Canadian NWP model from MSC Datamart. 4×/day at 00/06/12/18Z, 48h forecasts. Coverage capped at 60°N for v1 (SRTM stops there). HrdpsClient + scaffolding shipped 2026-04-29; full pipeline activates once the Rust prop-grid-rs HRDPS branch ships. Plan: docs/plans/2026-04-29-hrdps-canadian-prop-grid.md.
• ASOS (Automated Surface Observing System) — Surface weather via Iowa Environmental Mesonet.
• RAOB (Radiosonde) — Upper-air soundings via IEM.
• IEMRE — Gridded hourly reanalysis at 0.125° resolution.
• SRTM — 90m terrain elevation for path analysis (local tile cache or API fallback).
• Commercial Links — 7 microwave links near DFW polled via SNMP at 5-min intervals.

Commands

# Initial setup (deps, DB, assets)
mix setup

# Run dev server (live reloads automatically, no restart needed)
mix phx.server
iex -S mix phx.server    # with IEx shell

# Run all tests
mix test

# Run a single test file
mix test test/microwaveprop/terrain/terrain_analysis_test.exs

# Run previously failed tests
mix test --failed

# Format code (runs Styler plugin automatically)
mix format

# Static analysis
mix credo

# Pre-commit check (compile with warnings-as-errors, unlock unused deps, format, test)
mix precommit

# Rust pre-commit (matches the Forgejo CI step). Run from rust/prop_grid_rs/.
# `cargo build` alone does NOT catch clippy errors that gate the CI build.
cd rust/prop_grid_rs && cargo clippy --all-targets -- -D warnings && cargo test --release

# Database
mix ecto.create
mix ecto.migrate
mix ecto.gen.migration migration_name_using_underscores
mix ecto.reset           # drop + setup

# Run propagation grid manually
mix propagation_grid

# Assets
mix assets.build
mix assets.deploy        # minified + digest

Architecture

Directory Structure

• lib/microwaveprop/ — Business logic (contexts, schemas, workers)
  • propagation/ — Scoring engine: scorer.ex, band_config.ex, grid.ex, model.ex (ML), freshness_monitor.ex
  • weather/ — Data ingestion: hrrr_client.ex, iem_client.ex, sounding_params.ex, GRIB2 decoder
  • terrain/ — Path analysis: terrain_analysis.ex (ITU-R P.526-16), viewshed.ex, elevation_client.ex
  • radio/ — QSO schema, Maidenhead grid conversion
  • commercial/ — SNMP polling for commercial microwave links
  • workers/ — Oban background jobs for all data pipelines
• lib/microwaveprop_web/ — Web layer
  • live/map_live.ex — Main propagation map with forecast timeline
  • live/submit_live.ex — QSO submission form
  • live/algo_live.ex — Algorithm documentation page
  • live/qso_live/ — QSO list and detail views
  • plugs/remote_ip.ex — X-Forwarded-For client IP extraction
• config/ — Environment configs. runtime.exs has production Oban config (no backfill queues)
• assets/js/propagation_map_hook.js — Leaflet map, canvas heatmap, timeline, grid overlay
• algo.md — Full algorithm documentation with meteorological foundations
• priv/models/ — Saved ML model weights (gitignored)

Background Job Queues (Oban)

Queue	Workers	Purpose
propagation	PropagationGridWorker	Hourly HRRR fetch (f00-f18) + score computation
weather	WeatherFetchWorker	ASOS/RAOB data for QSO enrichment
hrrr	HrrrFetchWorker	HRRR profiles for individual QSOs
terrain	TerrainProfileWorker	ITU-R P.526-16 terrain diffraction analysis
iemre	IemreFetchWorker	IEMRE gridded weather for QSO enrichment
commercial	PollWorker	SNMP polling of commercial links
solar	SolarIndexWorker	Daily solar indices
enqueue	QsoWeatherEnqueueWorker	Batch enqueue enrichment jobs (dev cron only)
hrdps	HrdpsGridWorker	Canadian propagation chain seed (4×/day). Dormant until Rust prop-grid-rs HRDPS branch ships — module exists, cron entry deferred.

Production runs: propagation, commercial, solar, weather, hrrr, terrain, iemre queues. No cron backfill — enrichment is triggered by QSO submission only.

Dev runs the same hourly PropagationGridWorker + AsosAdjustmentWorker (10-minute nudge) cron as production.

Scoring Algorithm

10-factor weighted composite score (0-100) per grid point per band (recalibrated 2026-04-11 via gradient descent):

Factor	Weight	Source
Rain	13.6%	HRRR precipitation, ITU-R P.838-3
Humidity	12.4%	HRRR surface temp + dewpoint
PWAT	11.3%	HRRR precipitable water (column-integrated)
Season	11.1%	Month, band-dependent (inverted for 24G+)
Refractivity	10.5%	Native HRRR dM/dh (10-50m), fallback to pressure-level (250m)
Pressure	10.3%	HRRR surface pressure (frontal activity proxy)
T-Td depression	9.8%	HRRR surface
Sky cover	8.0%	HRRR cloud cover
Wind	8.0%	HRRR 10m wind
Time of day	5.0%	Solar-time adjusted diurnal cycle

Humidity effect reverses by frequency: beneficial at 10 GHz (refractivity), harmful at 24+ GHz (absorption).

Terrain Analysis (ITU-R P.526-16)

• Knife-edge diffraction loss (Eq. 31)
• Deygout 3-edge method for multiple obstacles
• Dynamic k-factor from HRRR refractivity gradient
• Profiles stored per QSO path (58k+ paths analyzed)

ML Model (Nx/Axon/EXLA)

Skeleton feed-forward network for future propagation prediction:

• 13 features: 8 atmospheric + 4 cyclical temporal + 1 log-frequency
• Architecture: Dense(64) → Dropout(0.2) → Dense(32) → Dropout(0.1) → Dense(1, sigmoid)
• Weights saved to priv/models/propagation_v1.nx
• Not yet trained — scaffolding only

Data Flow

HRRR (NOAA S3) → PropagationGridWorker (hourly, f00-f18)
  → store hrrr_profiles → compute scores → upsert propagation_scores
  → PubSub broadcast → MapLive → JS hook → canvas heatmap + timeline

User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers
  → enrich QSO with atmospheric + terrain data

Key Conventions

Phoenix / LiveView

• LiveView templates must start with <Layouts.app flash={@flash} ...> wrapping all content
• Use <.icon name="hero-x-mark" class="w-5 h-5"/> for icons, never Heroicons modules
• Use <.input> from core_components for form inputs
• Use to_form/2 for forms, never pass changesets directly to templates
• Use LiveView streams for collections, never phx-update="append"/"prepend"
• Avoid LiveComponents unless strongly justified
• LiveView names use Live suffix: MicrowavepropWeb.ThingLive
• Use <.link navigate={...}> / <.link patch={...}>, never live_redirect/live_patch
• Router scope aliases prefix automatically; don't add redundant aliases

HEEx Templates

• Use {...} for attribute interpolation, <%= %> only for block constructs (if/for/cond) in tag bodies
• Class lists must use [...] syntax for conditional classes
• Use <%!-- comment --%> for HTML comments
• Use phx-no-curly-interpolation on tags containing literal curly braces
• Never use <% Enum.each %>, always use <%= for item <- @collection do %>

JS / CSS

• Assets are bundled via esbuild and Tailwind mix tasks — never use npm, npx, or Node.js directly
• Tailwind v4: no tailwind.config.js, uses @import "tailwindcss" source(none) syntax in app.css
• Never use @apply in CSS
• No inline <script> tags; use colocated JS hooks with . prefix names
• Vendor deps must be imported into app.js/app.css, no external script src or link href
• LiveView hooks that push map_bounds (or any viewport-scoped data request) MUST implement reconnected() and a document.visibilitychange listener. Server assigns reset on reconnect while the hook keeps its stale client-side grid lookup — Leaflet tiles regenerated outside that extent paint blank. Both callbacks should call invalidateSize() + the bounds-push method; clean up the listener in destroyed().

Elixir

• Write idiomatic Elixir: use pattern matching, pipe operator, and the standard library
• Styler auto-formats on mix format (alias sorting, pipe chains, moduledoc enforcement, etc.)
• No index access on lists (mylist[i]); use Enum.at/2
• Bind results of if/case/cond blocks to variables (immutable rebinding)
• Never nest multiple modules in one file
• Use struct.field not struct[:field] (structs don't implement Access)
• Predicate functions: thing? not is_thing (reserve is_ for guards)
• Use Req for HTTP requests, never httpoison/tesla/httpc
• Always log anything that would normally be swallowed. Any place that drops {:exit, _} from Task.async_stream / Task.Supervisor.async_stream_nolink, ignores {:error, _} clauses, or otherwise discards a failure path MUST Logger.error/Logger.warning with enough context (input, key, reason) to debug from k8s logs. Silently dropping a tuple in a flat_map/reduce clause is a bug — production crashes inside an async task otherwise vanish (the path-calculator "0 / 9 HRRR points" outage was exactly this). LiveView start_async must have a matching handle_async(slot, {:exit, reason}, socket) clause that logs.

Ecto

• All schemas use @primary_key {:id, :binary_id, autogenerate: true}
• Preload associations in queries when accessed in templates
• field :name, :string for both string and text columns
• Use Ecto.Changeset.get_field/2 to access changeset fields
• Programmatic fields (e.g. user_id) must not be in cast calls
• Generate migrations with mix ecto.gen.migration
• Upserts via on_conflict: {:replace_all_except, [:id, ...]} with conflict_target

Testing

• Use start_supervised!/1 for process cleanup
• Use Process.monitor/1 + assert_receive {:DOWN, ...} instead of Process.sleep
• Use LazyHTML selectors for DOM assertions, never raw HTML matching
• Test against element IDs defined in templates
• Stub HTTP calls with Req.Test.stub in test setup
• Oban runs inline in test (testing: :inline)
• FreshnessMonitor disabled in test via config :microwaveprop, start_freshness_monitor: false
• Don't anchor relative-time test fixtures on DateTime.utc_now() truncated to the hour. A test that computes recent_t = (top_of_hour) - 30min is 30-90 min old of real utc_now() depending on what minute the test runs at; any production code that compares against utc_now() (e.g. a "drop entries older than 1 h" filter) flakes 50% of the time. Anchor at top_of_hour (or use a tiny offset like now - 1s) so the test is stable across clock minutes.

Rust (prop_grid_rs)

• Always run cargo clippy --all-targets -- -D warnings before committing Rust changes. The Forgejo CI step runs clippy with warnings-as-errors and a passing local cargo build does NOT catch lints (e.g. manual_range_contains, iter_kv_map, useless_format). A clippy failure blocks the image build and the deploy. The Rust precommit recipe is in the Commands section.
• The CI build pipeline runs cargo clippy --all-targets -- -D warnings && cargo test --release && cargo build --release. Match this locally — cargo build alone is not enough.

/about page maintenance

• Refresh lib/microwaveprop_web/live/about_live.ex whenever algorithm structure, factor count or weights, major data sources, schema scale claims (e.g. "58k+ contacts"), or roadmap items change. The dashboard counts auto-refresh from the DB; the prose blocks (overview, "How it works", "How it's built", "What's next") do not.
• The page renders a per-table count via count_exact/1 and count_estimate/1. Each query is wrapped individually so a missing dev table doesn't blank the whole dashboard — keep that pattern when adding new stats.

Deployment

• Production runs on Kubernetes (home-cluster context, prop namespace) — use kubectl -n prop ...
• Deployment name prop (3 replicas). Inspect with kubectl -n prop get pods, kubectl -n prop logs deploy/prop, kubectl -n prop exec deploy/prop -- ...
• Push to github remote for code. The dokku remote is obsolete — do not push there.
• Production Oban config in config/runtime.exs — no backfill cron, enrichment triggered by QSO submission
• Shared NFS mount /data in production container (server: node3 at 10.0.15.103:/data). SRTM tiles live at /data/srtm.