System.cmd :timeout option doesn't exist in Elixir 1.19, causing every
poll to crash silently. Also translate SNMPv2-SMI::enterprises prefix
to numeric OID so the lookup maps match correctly.
Weather job fan-out can exceed PostgreSQL's 65535 parameter limit
when many QSOs produce thousands of jobs in a single batch. Chunk
insert_all into batches of 1000 jobs.
- Add precip_1h_in and wx_codes fields to ASOS surface observations
- Add IEMRE reanalysis schema for radar-derived hourly precipitation
- Add IemreFetchWorker with exponential backoff and idempotency
- Integrate IEMRE enqueue into cron weather backfill pipeline
- All existing QSOs marked iemre_queued=false for automatic backfill
Poll UBNT AirFiber radios (AF11X + AF60-LR) every 5 minutes via
net-snmp CLI, storing signal metrics in commercial_samples. Fetches
ASOS weather alongside each cycle for propagation correlation.
Includes 7 seeded link definitions, Oban cron worker, and net-snmp
in the Docker image.
Previously each run only enqueued 500 QSOs (the query limit).
Now weather, HRRR, and terrain enqueue functions recurse until
no unprocessed QSOs remain.
extract_n_values_array used Enum.take(count) on a reversed list
before reversing it, which included padding values from byte
alignment and dropped actual values. When group count * bits
wasn't a multiple of 8, the extra padding bits produced a
phantom value that shifted the entire array by one position.
This caused cascading errors in spatial differencing — values
started correct but diverged exponentially (DPT decoded as
38 billion K instead of 275 K).
Fix: reverse the list first, then take count, so padding values
at the end are discarded instead of actual values at the start.
AWS S3 doesn't support multi-range HTTP requests. When given
Range: bytes=0-999, 2000-2999 it ignores the header and returns
the full file (200 instead of 206). Download each range separately.
Instead of listing every permanent failure string, only retry on known
transient errors (5xx, 429, network exceptions). Everything else —
GRIB decode errors, 404s, range request failures, index out of range —
is cancelled immediately.
S3 returns 200 with the full ~116MB file instead of 206 for some older
HRRR data. Treat this as a permanent failure instead of trying to parse
the entire file.
Instead of returning "malformed section" on trailing bytes or truncated
sections, attempt to return parsed results. This handles older HRRR
files (2019) that have padding bytes after the last section.
When a local .hgt tile is missing, download it from the public AWS S3
skadi bucket, decompress with zlib, and write to the tiles directory
before retrying the lookup. Falls back to Open-Meteo/OpenTopo APIs if
the download fails.
HTTP 404 means the HRRR data doesn't exist on NOAA S3 (pre-2014
dates, etc.) and will never succeed. Return {:cancel, reason}
instead of {:error, reason} so Oban stops retrying immediately.
consume_bits would crash with a MatchError when the remaining
bitstring had fewer bits than the group width. Added a guard
clause to stop consuming when insufficient bits remain, matching
how wgrib2 handles trailing padding in packed data sections.
Eliminates the external wgrib2 C tool dependency that blocked HRRR
processing. Implements Lambert Conformal projection, simple packing
(Template 5.0), complex packing with spatial differencing (Template 5.3),
and GRIB2 section parsing — enough to extract point values from HRRR
grid data using only Elixir.
Previously retry: :transient only retried connection errors, not HTTP
429 rate limits. All clients now use a custom retry function that
handles 429, 500, 502, 503, 504 with 5 retries and jittered backoff.
- Defensive grid-to-coordinate resolution in create_qso (case instead of crash on match failure)
- Log warning when Oban weather enqueue fails instead of silently dropping
- Add 30s per-session submission cooldown to prevent rapid-fire submissions
20 attempts with exponential backoff capped at 6 hours ensures
data is eventually fetched even during extended rate limiting or
API outages (~3 days of retrying before giving up).
Maidenhead grid module converts grid squares to lat/lon coordinates.
Submission form validates grids, bands, modes, and email, computes
positions and distance, then triggers the weather/HRRR/terrain
processing pipeline via Oban.
Fetch elevation data along the path between two stations via the
Open-Meteo Elevation API (with Open-Topo-Data fallback), compute
Fresnel zone clearance, earth bulge, and knife-edge diffraction loss,
and store the results per QSO.
- terrain_profiles table with migration and TerrainProfile schema
- ElevationClient with batched API calls and fallback
- TerrainAnalysis with Fresnel/diffraction physics (ITU-R P.526-15)
- TerrainProfileWorker on Oban :terrain queue
- QsoWeatherEnqueueWorker enqueues terrain jobs automatically
- QSO show page displays verdict badge and collapsible elevation table
- Reorder show page: terrain, soundings, solar, HRRR, surface obs
- Fix Dockerfile wgrib2 build (add cmake dependency)
HRRR provides hourly 3km-resolution atmospheric profiles, filling the
temporal gaps (12-hourly soundings) and spatial gaps (only 9 sounding
stations) in our current weather data.
- Add hrrr_profiles table and hrrr_queued flag on QSOs
- HrrrClient fetches GRIB2 data via HTTP Range requests + wgrib2
- HrrrFetchWorker derives refractivity/ducting params via SoundingParams
- QsoWeatherEnqueueWorker now also enqueues HRRR jobs
- QSO show page displays HRRR section with collapsible profile
- Dockerfile builds wgrib2 from source for production
Fix pos1 longitude key mismatch (lng vs lon) that caused empty weather
results for all QSOs. Bump search radius to 300km for sounding coverage.
Soundings now show as expandable cards with full elevation profile data.
Adds haversine_km/4 to Radio context and backfill_distances/1 which
calculates distance_km from pos1/pos2 for QSOs missing it. Called
by QsoWeatherEnqueueWorker before enqueueing weather fetches.
Oban cron worker runs every 4 hours, finds QSOs without weather data,
discovers nearby ASOS/sounding stations, and enqueues WeatherFetchWorker
jobs. Migrations run automatically on app start in production.
IEM was returning 503s under 10 concurrent workers. Three fixes:
- Req transient retry with exponential backoff on IEM requests
- WeatherFetchWorker Oban job retries failed ASOS/RAOB fetches 2-3h later
with random jitter to avoid thundering herd
- Import script concurrency reduced 10→5, weather queue capped at 3
Store surface observations (ASOS) and upper-air soundings (RAOB) alongside
QSOs for atmospheric propagation correlation. Three new tables: weather_stations,
surface_observations, and soundings with JSONB profiles and pre-computed derived
parameters (refractivity, gradients, duct detection, stability indices).
Includes IEM API client for historical data import and import script seeded
with 95 ASOS + 9 sounding stations from PropCast coverage area.
