ApiDocsController + ApiDocsLive read docs/api/{README.md,openapi.yaml}
at compile time and bake the contents into module attributes. The
build context excluded those files because nothing in the Dockerfile
copied the docs directory, so `mix compile` failed during the image
build.
Move the runtime base image from git.mcintire.me/graham/prop-base
to codeberg.org/gmcintire/prop-base. The shared REGISTRY_USER /
REGISTRY_PASSWORD secrets are pointed at Codeberg credentials;
the build-base workflow hardcodes env.REGISTRY=codeberg.org so it
can't accidentally push to a different registry if the URL secret
drifts.
The main app + prop-grid-rs builds still pull this base via the
BASE_IMAGE arg in Dockerfile, now codeberg.org/gmcintire/prop-base:latest.
emqtt's transitive `quicer` dep needs CMake + msquic submodule
clones to build, which the slim Debian builder image doesn't ship —
the prod CI build was failing inside `mix deps.compile`. Tortoise311
is the obvious pure-Elixir alternative but it bundles a
gen_state_machine + retry/inflight stack we don't use because we're
subscribe-only at QoS 0.
Implementing the wire protocol inline is ~150 lines in
`Microwaveprop.Pskr.Mqtt` (CONNECT/SUBSCRIBE/PINGREQ/DISCONNECT
builders, CONNACK/PUBLISH/SUBACK/PINGRESP parser, varint codec).
Pskr.Client now uses :gen_tcp with `active: :once` framing,
buffers partial reads, schedules its own keepalive, and parses
inbound packets in a tight loop.
Drops `BUILD_WITHOUT_QUIC=1` from the Dockerfile — no longer
relevant since there's no native dep at all in the build chain
now.
Subscribes to mqtt.pskreporter.info:1883 over plain TCP and folds
every reception report into hourly aggregates per
(band, sender_grid_4, receiver_grid_4). Each spot is the empirical
"propagation actually occurred on this path right now" signal we'll
correlate against HRRR atmospheric state to recalibrate the scoring
weights. Aggregating at the path-hour bucket collapses 50-reporter
QSOs to one row instead of 50.
Topic filter anchors on USA (DXCC 291) on either sender or receiver
side so the broker pre-filters out everything outside our HRRR
domain. Bands subscribed: VHF and up (6m, 2m, 70cm, 23cm, +
microwave) — HF is dominated by ionospheric propagation, which this
project doesn't model.
Pskr.Client cluster-elects a singleton via :global.register_name —
all replicas start the GenServer but only one connects to MQTT;
the rest stay in :standby and watch for the leader's nodedown to
re-run election. Off in dev/test, on by default in prod
(PSKR_MQTT_ENABLED=false as kill-switch).
Pskr.Aggregator buffers in memory keyed by Pskr.path_key/1 and
flushes every 60s via Repo.insert_all/3 with an additive
ON CONFLICT (count summed, SNR envelope by GREATEST/LEAST,
modes unioned via unnest+DISTINCT). Idempotent across overlapping
flushes and across leader handover.
Dockerfile sets BUILD_WITHOUT_QUIC=1 — emqtt's transitive quicer
dep wants CMake + OpenSSL headers we'd otherwise have to add to
the builder image just to never use QUIC over plain MQTT. Base
image is unchanged; the new dep compiles cleanly into the existing
prop-base runtime.
The wgrib2-builder stage and the runtime apt-install stages
(libstdc++6/openssl/locale + cdo/gdal-bin) now live in
git.mcintire.me/graham/prop-base, built separately by build-base.yaml.
The app Dockerfile FROMs that image and just COPYs the Elixir release
on top.
Net effect: typical pushes drop ~8 min of wgrib2 compile + ~2 min of
cdo/gdal-bin apt installs. Bumping wgrib2 or g2c is now a one-line
change to Dockerfile.base — the next push picks up :latest
automatically.
wgrib2 3.8.0's CMakeLists.txt:93 calls find_package(g2c 2.3.0), so the
previous 2.1.0 pin makes both Dockerfiles error out with "Could not find
a configuration file for package g2c that is compatible with requested
version 2.3.0". Bumping to 2.3.0 across both the Elixir and Rust images
since their wgrib2-builder stages stay in lockstep.
Production wgrib2 3.8.0 still hit `*** FATAL ERROR: packing type 40
not supported ***` on HRDPS files. ldd confirmed wgrib2 was linking
only libc/libm — no OpenJPEG, no Jasper, no g2c.
wgrib2 3.8.0 cmake doesn't have USE_OPENJPEG or USE_JASPER options.
The legacy flag names we passed were silently ignored. The actual
knob is USE_G2CLIB_LOW, which links wgrib2 to the external g2c lib
(already built in the previous stage with OpenJPEG support). Dropped
the bogus USE_PNG / USE_OPENJPEG flags and added USE_G2CLIB_LOW=ON
in both Dockerfiles.
Verified by reading wgrib2 v3.8.0's CMakeLists.txt: the documented
JPEG2000 path is "USE_G2CLIB_LOW: Use g2c low-level decoders
(png,jpeg2000)?".
Reproduced in production pod 2026-04-30: wgrib2 3.6.0 crashes with
`free(): invalid size` when -lon point extraction is run against any
HRDPS rotated lat/lon GRIB2 file, regardless of batch size (failed at
N=200, N=1000, N=2000). Output before the abort is denormal f32 garbage
(val=4.93944e-41) — wgrib2's internal record buffer is being corrupted.
Same files extract cleanly with wgrib2 3.8.0 locally. Bumping the build
arg in both Dockerfile (Elixir image, used by hrrr-point-rs/per-QSO
HRDPS lookups) and rust/prop_grid_rs/Dockerfile (Rust grid worker) so
both consumers get the fix.
Also bumped POINT_BATCH 200 → 1000 in decoder.rs. With wgrib2 3.8.0
stable at 1000+ point batches, this drops the chain-step invocation
count from 286 → 57 per step. Comment captures the reproduction
context so a future bisection finds it quickly.
Rover ranking now applies a per-cell tree-canopy clutter penalty
(1 dB / 8 m, capped at 4 dB) so cells in dense forests get down-ranked
even when their per-station path clearance survives. PathTerrain
already accounts for trees ON the link path; this surfaces "I'm in a
forest, every direction is foliage."
Canopy.BulkFetch downloads Lang et al. 2020 10 m global canopy-height
3-degree COGs and slices each into 1° × 1° uint8 .canopy tiles via
gdal_translate, the runtime image now ships gdal-bin. Run on prod via:
bin/microwaveprop rpc 'Microwaveprop.Canopy.BulkFetch.run(32.8, -97.0, 200)'
Tile cache lives under /data/canopy/. Already-sliced tiles are skipped.
The last attempt still blew past the CI runner's /var cap because
apt's pre-download free-space check runs against the archives dir,
and the image overlay is where that sits. Moving /var/cache/apt and
/var/lib/apt onto BuildKit cache mounts puts the .deb storage on the
builder's cache volume (outside the image overlay), so the free-space
check runs against whatever disk backs the cache — which on the CI
runner has room.
Requires the `syntax=docker/dockerfile:1.6` directive at the top for
the --mount flag. Also removes /etc/apt/apt.conf.d/docker-clean so
the Debian base's post-invoke hook doesn't wipe the cache mount on
every apt run.
The cdo install pulls ~1 GB of transitive deps (netCDF/HDF5/ecCodes/
Fortran) and was exhausting CI's /var/cache/apt partition. Two fixes:
- `APT::Keep-Downloaded-Packages=false` deletes each .deb immediately
after it's extracted, so the archive cache doesn't retain every
downloaded package through the end of the install.
- Split cdo into its own RUN layer so the base runtime deps finish
installing and clean before cdo's cache starts growing.
Docker final stage bakes BUILD_TIMESTAMP into /app/BUILD_TIMESTAMP via
build-arg (only consumed after mix compile, so earlier layer caching is
preserved). Application.build_timestamp/0 reads the file, falls back to
DEPLOY_TIMESTAMP env, then to compile time. Navbar renders a compact
"Deployed Xm ago" with the full UTC time in the tooltip.
AgentSkillsController reads priv/agent-skills/*.md at compile time via
@external_resource + File.read! to bake sha256 digests into the module.
priv/ is copied after compile to keep the compile cache stable, so add
a targeted COPY for the agent-skills subdir before compile — same
pattern as algo.md.
The linker error from the previous attempt was actual symbol
references, not a flag-combination oddity:
undefined reference to `g2c_enc_jpeg2000'
undefined reference to `g2c_dec_jpeg2000'
Those live in g2c's "new" file-based API, built only when
BUILD_G2C=ON. In v1.9.0 the option defaulted OFF and was marked
"experimental until 2.0.0 release", so I had explicitly disabled
it. In v2.1.0 (released 2025-01-13, before wgrib2 v3.6.0 on
2025-02-19) it defaults ON and the API is stable. Bump
G2C_VERSION to 2.1.0 and drop the BUILD_G2C override.
With the new-API symbols now exported, wgrib2 can be compiled
with every decoder we care about in one shot:
-DUSE_PNG=ON -DUSE_OPENJPEG=ON -DUSE_AEC=ON
`wgrib2 -config` in the final runtime image now reports:
Supported decoding: simple, complex, rle, ieee, png, jpeg2000,
CCSDS AEC
Verified end-to-end by building the full image with podman and
running wgrib2 -config inside it. MrmsFetchWorker's packing
type 41 (PNG) messages will decode once the rolling deploy lands.
Last attempt (-DUSE_PNG=ON -DUSE_AEC=ON on wgrib2) reached 100 %
linking libwgrib2.so and then died with "make: Error 2" on the
parallel wgrib2_exe link. wgrib2's own CI workflow
(.github/workflows/Linux_options.yml) never tests USE_PNG alone —
its matrix uses -DUSE_OPENJPEG=ON, and its NCEPLIBS-g2c is built
with -DUSE_OpenJPEG=ON -DUSE_Jasper=OFF. Copying that known-good
combination:
* g2c now builds with USE_PNG + USE_AEC + USE_OpenJPEG (Jasper
still disabled — libjasper is out of Debian anyway).
* wgrib2 flips to -DUSE_OPENJPEG=ON -DUSE_AEC=ON.
* libopenjp2-7-dev added to the builder apt list, libopenjp2-7
added to the runtime apt list so libg2c.so + wgrib2 can
resolve OpenJPEG symbols at exec time.
wgrib2 delegates actual GRIB2 decompression to g2c, which is
built with PNG support, so DRT 5.41 (MRMS PrecipRate) still
decodes through this link — wgrib2's own USE_PNG knob isn't
required for decoding, only for encoding we don't do.
The previous attempt to flip `-DUSE_PNG=ON -DUSE_AEC=ON` on wgrib2
broke the image build because wgrib2 v3.6.0's CMakeLists.txt
line 94 forces `find_package(g2c 1.9.0 REQUIRED)` whenever PNG or
JPEG2000 is enabled, and NCEPLIBS-g2c wasn't installed in the
wgrib2-builder stage.
Install NCEPLIBS-g2c v1.9.0 (the version wgrib2 asks for) in the
wgrib2-builder stage ahead of wgrib2 itself, with:
- USE_PNG=ON (needs libpng-dev, already present)
- USE_AEC=ON (needs libaec-dev, already present)
- USE_Jasper=OFF (default ON — would need libjasper-dev
and another runtime dep we don't need)
- USE_OpenJPEG=OFF (same rationale)
- BUILD_G2C=OFF (new experimental API, not needed here)
Then flip wgrib2's PNG/AEC flags back on. wgrib2's binary links
dynamically against libg2c.so so the final stage copies
/usr/local/lib/libg2c.so* from the builder and refreshes the
loader cache with ldconfig.
With the decoder restored, re-add MrmsFetchWorker to the prod
cron in runtime.exs. That worker pulls NOAA MRMS PrecipRate grids
every 2 minutes, which use GRIB2 DRT 5.41 (PNG-compressed) — the
whole reason we needed the PNG decoder in the first place. ASOS
nudges will now have a live radar overlay again.
The previous commit that added -DUSE_PNG=ON -DUSE_AEC=ON to the
wgrib2 cmake invocation broke the image build: wgrib2 3.6.0 turns
find_package(g2c 1.9.0 REQUIRED) on when those flags are enabled,
and NCEPLIBS-g2c isn't installed in the wgrib2-builder stage.
Revert to the stock wgrib2 cmake config so CI is green again and
pull MrmsFetchWorker out of the prod cron instead. Without the
PNG decoder the worker can't read MRMS PrecipRate (GRIB2 DRT
5.41) and was logging "packing type 41 not supported" every two
minutes. AsosAdjustmentWorker's MRMS path already handles an
empty cache gracefully — the ASOS nudge still runs, it just
drops the live radar overlay.
Re-add MrmsFetchWorker once the Dockerfile installs NCEPLIBS-g2c
and rebuilds wgrib2 with PNG/AEC support.
Prod wgrib2 3.6.0 was built without PNG support (cmake had no
-DUSE_PNG flag) so MrmsFetchWorker crashed on every tick with
'packing type 41 not supported' — MRMS PrecipRate tiles are
PNG-compressed (GRIB2 DRT 5.41). libpng-dev was already in the
builder stage; the cmake invocation just never asked wgrib2 to
use it.
Also turn on -DUSE_AEC=ON since libaec-dev is already installed
and some HRRR native-grid variables ship with AEC/CCSDS 121
compression.
Vendor oban_web 2.12.1 and oban_met 1.1.0 (taken from the
towerops-web2 vendor tree), bump oban to ~> 2.21, and mount the Oban
dashboard at /admin/oban behind the existing require_admin on_mount
hook. Adds an admin-only "Oban" nav link and extends the Dockerfile
to copy vendor/ before deps.get so production builds pick up the
path dependencies.
Move COPY priv after mix compile so ML model retraining doesn't bust
the compile cache. Create empty priv dirs before compile for
:code.priv_dir. Move algo.md after compile. Add docs/plans and
analysis outputs to .dockerignore.
Add a separate wgrib2-builder stage that compiles wgrib2 from source.
Docker layer caching means this only builds once — subsequent deploys
reuse the cached layer. The binary is copied into the final runtime
image. Also fixed wgrib2 path lookup to be runtime instead of
compile-time so it works in the Docker build pipeline.
Poll UBNT AirFiber radios (AF11X + AF60-LR) every 5 minutes via
net-snmp CLI, storing signal metrics in commercial_samples. Fetches
ASOS weather alongside each cycle for propagation correlation.
Includes 7 seeded link definitions, Oban cron worker, and net-snmp
in the Docker image.
Fetch elevation data along the path between two stations via the
Open-Meteo Elevation API (with Open-Topo-Data fallback), compute
Fresnel zone clearance, earth bulge, and knife-edge diffraction loss,
and store the results per QSO.
- terrain_profiles table with migration and TerrainProfile schema
- ElevationClient with batched API calls and fallback
- TerrainAnalysis with Fresnel/diffraction physics (ITU-R P.526-15)
- TerrainProfileWorker on Oban :terrain queue
- QsoWeatherEnqueueWorker enqueues terrain jobs automatically
- QSO show page displays verdict badge and collapsible elevation table
- Reorder show page: terrain, soundings, solar, HRRR, surface obs
- Fix Dockerfile wgrib2 build (add cmake dependency)
HRRR provides hourly 3km-resolution atmospheric profiles, filling the
temporal gaps (12-hourly soundings) and spatial gaps (only 9 sounding
stations) in our current weather data.
- Add hrrr_profiles table and hrrr_queued flag on QSOs
- HrrrClient fetches GRIB2 data via HTTP Range requests + wgrib2
- HrrrFetchWorker derives refractivity/ducting params via SoundingParams
- QsoWeatherEnqueueWorker now also enqueues HRRR jobs
- QSO show page displays HRRR section with collapsible profile
- Dockerfile builds wgrib2 from source for production
