diff --git a/content/_index.md b/content/_index.md index 7e243d4..b43fba8 100644 --- a/content/_index.md +++ b/content/_index.md @@ -2,4 +2,4 @@ title = "W5ISP" +++ -I'm Graham McIntire (W5ISP), a site reliability and security engineer based in Blue Ridge, Texas. I like building things and figuring out how they work. Licensed amateur radio operator since 1996 and plain text enthusiast. +I'm Graham McIntire (W5ISP), a site reliability and security engineer based in Blue Ridge, Texas. I like building things and figuring out how they work. Licensed amateur radio operator since 1996 and plain text enthusiast. [I'm looking for work, please hire me!](https://linkedin.com/in/grahammcintire) diff --git a/content/blog/postgres1.md b/content/blog/postgres1.md index 7867bc3..9990710 100644 --- a/content/blog/postgres1.md +++ b/content/blog/postgres1.md @@ -1,9 +1,11 @@ +++ title = "When Postgres isn't the right choice" -date = 2026-04-14 +date = 2026-04-18 draft = true +++ +I've been working on For the [Microwave Propagation](https://prop.w5isp.com) site I've been working on + ┌────────────────────────────────────────┬───────────┬───────────┐ │ Phase │ Wall time │ % of fh=0 │ ├────────────────────────────────────────┼───────────┼───────────┤ @@ -21,3 +23,80 @@ draft = true ├────────────────────────────────────────┼───────────┼───────────┤ │ Total fh=0 │ 562s │ 100% │ └────────────────────────────────────────┴───────────┴───────────┘ + + The propagation map at prop.w5isp.com ingests HRRR weather every hour and re-scores the entire CONUS grid across five + amateur-radio microwave bands. That is 95,000 grid points × 5 bands × 19 forecast hours = ~9 million score values per chain. For + a long time those lived in a Postgres table called propagation_scores. They don't anymore, and the map went from sluggish to + instantaneous. This is the story of why. + + The shape mismatch + + Each hourly run did this: + + DELETE FROM propagation_scores WHERE valid_time = $1; + INSERT INTO propagation_scores (lat, lon, band_mhz, valid_time, score, factors) ... + + 475,000 rows per forecast hour, funneled through four indexes (PK UUID, uniqueness tuple, valid_time, (band_mhz, valid_time)), + plus a JSONB factors blob for the analysis hour. Scoring + upsert took about 4m 40s per forecast hour. Across 19 hours that's + more than an entire hourly cron cycle, so we had to back off to every-three-hours just to keep up. + + The problem wasn't Postgres — Postgres was doing exactly what we asked. The problem was that the shape we were asking for was + wrong for how the data gets read. + + The map is a canvas heatmap. It wants one dense (lat, lon) → score array for a single (band, valid_time). We were storing it as + half a million row tuples with MVCC bookkeeping, toast pages, and a B-tree index per column we'd never query by. Every write + rematerialized metadata the reader throws away. + + Three stopgaps that weren't enough + + Before rewriting the store, we tuned what was there: + + 1. DELETE + INSERT … with no ON CONFLICT. The chain worker rewrites a full (valid_time, all bands) slice every hour, so conflict + detection was pure waste. Skipping it shaved a chunk off. + 2. Skip factors on forecast hours. factors is a ~200-byte JSONB per row describing the ten scoring components (rain, humidity, + refractivity, etc.). We only show it on the analysis hour (f00) when the user clicks a cell. Passing factors: nil for f01–f18 + skips a JSONB encode plus a toast write for roughly half the volume. + 3. UNLOGGED table. Writes bypass the WAL entirely. On unclean shutdown the table truncates — fine, because PropagationGridWorker + rebuilds it from HRRR every hour. Durability was never the point; this was a cache with extra steps. + + These helped, but the per-row overhead of a row-oriented store against a dense numeric grid is a ceiling you can't tune past. The + phase was still the wall-clock dominator. + + + The binary file + + Each (band_mhz, valid_time) grid now lands in one file at /data/scores/{band}/{iso}.ntms: + + magic : 4 bytes "NTMS" + version : 1 byte 0x01 + band_mhz : 4 bytes uint32 LE + valid_time : 8 bytes int64 unix seconds LE + lat_min : 4 bytes float32 LE + lon_min : 4 bytes float32 LE + step_deg : 4 bytes float32 LE + n_rows : 2 bytes uint16 LE + n_cols : 2 bytes uint16 LE + scores : n_rows × n_cols bytes (uint8, 0–100; 255 = no-data) + + 33-byte header, dense uint8 array. A full CONUS grid serializes to ~93 KB per band per hour. A cell at (lat, lon) is at byte + offset row × n_cols + col, so a point lookup is arithmetic plus a single File.read/1. No index, no query planner, no lock + manager. + + Writes go through the temp-then-rename pattern: + + tmp = path <> ".tmp." <> unique_suffix() + File.write!(tmp, binary, [:binary]) + File.rename!(tmp, path) + + With Postgres off the hot path, the full f00–f18 chain dropped from ~170 min to ~45–60 min, so we ran the cron up to hourly. The + 15-minute prune cron reaped expired files with rm — instant. + + The f00 factor breakdown + + One thing binary files don't handle well is the analysis-hour factor breakdown (the 10-component panel you see when you click a + cell). Those factors are heterogeneous Elixir maps, not a numeric grid. + + The solution was a second file format, ProfilesFile: one compressed ETF file per valid_time at + /data/scores/profiles/{iso}.etf.gz, storing the enriched HRRR grid keyed by {lat, lon}. On a click, we reload the relevant cell's + profile and rescore on demand. Same atomic-rename write pattern, different payload — ETF where heterogeneity matters, raw binary + where density does. diff --git a/content/blog/prop1.md b/content/blog/prop1.md new file mode 100644 index 0000000..0cb1708 --- /dev/null +++ b/content/blog/prop1.md @@ -0,0 +1,71 @@ ++++ +title = "How can microwave contacts be made non-LOS?" +date = 2026-04-18 +draft = false ++++ + +How can microwave contacts be made non-LOS is a question I've had for a while now. +From running a WISP I'm very familiar with point-to-point microwave links. However, +amateur radio operators have been making NLOS contacts for years when they technically +shouldn't work at all. + +As an example, this is the evelation path between my QTH and an [NTMS beacon on top of Texas +Women's University's dorm building](https://prop.w5isp.com/beacons/dddca785-c3e7-45f8-8f9f-edc748f881d6): + +![Path profile](/images/path_profile.png) + +This lead to a deep rabbit hole to investigate why contacts work some times and HOW they work. +There are already known propagation methods such as EME (moonbounce) or rain / meteor scatter, +however it's also possible to form ducts within the lower levels of the atmosphere with favorable +conditions. An easy to understand demonstration is why sometimes ships appear like they're floating: + +![Atmospheric ducted ship](/images/atmospheric_ducting.jpg) + +This is because there's a duct with favorable conditions for refracting visible light -- the same +thing happen for microwave. + +At one of the [NTMS](https://ntms.org) meetings, KM5PO presented about this same thing and had used +claude to write out some client-side pages to help predict and analyze the propagation. It was based +on limited data, but did an okay job guessing at why the propagation worked. + +Fast forward a few weeks and I've now build an elixir/phoenix app at that +collects and processes everything. For a quick import of some known contacts made, I imported all of the ARRL 10 GHz and up contests as well as the 222 MHz and up contests. + +With known contacts in place, it was time to backfill them with data, including: + +### Atmospheric +* HRRR (NOAA 3 km) — hourly f00–f18 propagation scoring hot path; native hybrid-sigma levels for fine-grained duct detection +* GEFS (NOAA 0.5° ensemble) — Day 2–7 extended-horizon outlook +* HRRR native (hybrid σ) — best-duct-band per cell, refractivity gradient at 10–50 m resolution +* IEMRE (Iowa State 0.125° reanalysis) — gridded hourly weather for per-QSO enrichment +* NARR (NCEI 32 km reanalysis) — pre-2014 historical backfill for old QSOs (HRRR archive doesn't cover them) + +### Surface observations +* ASOS (via IEM) — airport station surface weather, used by per-QSO enrichment and the 10-minute score nudge +* MRMS (NOAA PrecipRate) — 2-minute radar-derived rain rate that overlays HRRR's hourly precip for fast-moving cells +* NEXRAD (Level II composite reflectivity) — current-hour precip cell detection for f00 scoring + +### Upper air +* RAOB (US radiosondes via IEM) — twice-daily soundings for duct/inversion analysis +* UWYO (Canadian radiosondes) — 00/12 Z Canadian profiles + +### Terrain & geography +* SRTM (90 m local tile cache + fallback API) — ITU-R P.526-16 path diffraction + viewshed + +It's taken a mountain of work and time to get all of this implemented, but most of the contacts +have now been backfilled with this information. Feeding all of this in to Claude, we end up with an +[algorithm](https://prop.w5isp.com/algo) that understands and can predict future conditions based +on weather data. + +This algorithm gets turned in to code, and then we can fetch HRRR data hourly, ammend with ASOS +when avaliable and feed it through the algorithm to determine if any given area's atmosphere is +favorable to propagation. Our predictions are still estimated and may not have any bearing on +reality. + +Now our mission is to continue collecting as many verified contacts on 50 MHz and up and enrich +to refine our prediction model gradually over time. Another sub-project is automated beacon +monitoring. Any NLOS beacon that can be automatically monitored continuously provides an enormous +amount of data to train our model with since both points are known. + +If you'd like to help out, we need as many contacts during non-contest times as possible and future +automated beacon monitoring. You can reach me at my first name at my last name dot me (it's on the front page.) diff --git a/content/projects/ntmsprop.md b/content/projects/ntmsprop.md new file mode 100644 index 0000000..7a26210 --- /dev/null +++ b/content/projects/ntmsprop.md @@ -0,0 +1,15 @@ ++++ +title = "NTMS Microwave Propagation" +description = "Microwave propagation prediction based on current and future weather" + +[extra] +url = "https://prop.w5isp.com" ++++ + +This site started as a curiosity I've had for a while: why do some amateur radio contacts work non-line of sight on microwave frequencies when they shouldn't. + +It's evolved in to a community effort along with the [North Texas Microwave Society](https://ntms.org) including from a senior meteorologist in Lubbock, Tx. + +Hourly we collect [HRRR](https://rapidrefresh.noaa.gov/hrrr/) data for the entire continental USA, process it, then store the estimated propagation in small binary files for display. + +Tech stack is elixir, phoenix, postgres, k8s. diff --git a/sass/overrides.scss b/sass/overrides.scss new file mode 100644 index 0000000..d6eca11 --- /dev/null +++ b/sass/overrides.scss @@ -0,0 +1,8 @@ +h1:before, +h2:before, +h3:before, +h4:before, +h5:before, +h6:before { + content: none; +} diff --git a/static/images/atmospheric_ducting.jpg b/static/images/atmospheric_ducting.jpg new file mode 100644 index 0000000..9584521 Binary files /dev/null and b/static/images/atmospheric_ducting.jpg differ diff --git a/static/images/path_profile.png b/static/images/path_profile.png new file mode 100644 index 0000000..e9c4a13 Binary files /dev/null and b/static/images/path_profile.png differ diff --git a/templates/base.html b/templates/base.html index 6719be1..abfc90f 100644 --- a/templates/base.html +++ b/templates/base.html @@ -24,6 +24,7 @@ {% else %} {% endif %} + {% if config.extra.zolanight.google_analytics_id %} +