bulk_richardson, theta_e_jump, and shear_at_top feature functions
pull derived fields from the nearest hrrr_native_profile. Ready for
mix backtest once sufficient data is backfilled.
Inversion detection module (Propagation.Inversion):
- find_inversion_top/1 walks the native profile to locate the first
temperature inversion (surface-based or elevated)
- bulk_richardson/3 computes the Richardson number across the
inversion layer (Ri < 0.25 = turbulent, > 1 = laminar/good)
- shear_magnitude/3 computes the wind shear vector magnitude
- potential_temperature/2 for θ = T*(P0/P)^0.286
Theta-e module (Weather.ThetaE):
- Bolton (1980) equivalent potential temperature
- dewpoint_from_spfh/2 via Magnus-Tetens inversion
- theta_e_jump/3 for the thermodynamic decoupling metric
mix hrrr_native_derive_fields populates inversion_top_m,
bulk_richardson, theta_e_jump_k, and shear_at_top_ms on existing
hrrr_native_profiles rows.
First real data: 2022-08-20 12Z TX profile shows inversion at
186 m, Ri = 0.16 (turbulent), θ_e jump = 0.33 K — consistent with
marginal propagation conditions at that hour.
wgrib2 -lola ... bin writes Fortran unformatted records (4-byte
length header + data + 4-byte length trailer per message).
parse_lola_binary was treating the binary as tightly packed,
causing every message after the first to read from the wrong
offset — values came out as garbage across all grid points.
Fix: account for the 8-byte record overhead per message when
computing the data offset for each message's grid values.
This bug affects both the existing propagation grid extraction
(which may have been producing subtly wrong scores) and the new
native-level extraction (which was producing obviously wrong
values). The fix is a one-line stride change.
Also adds Backtest.Features.native_surface_refractivity for the
Phase 1 sanity check, plus a tighter wgrib2 match pattern that
selects only hybrid-level messages from the native file.
The Elixir GRIB2 decoder didn't map level type 105 (hybrid) or
variable IDs for SPFH and TKE, so native-level messages decoded as
"unknown:105:N" keys that build_native_profile couldn't find. Add
the three missing mappings to Section.identify_level/identify_var.
Also add HrrrNativeClient.extract_native_profiles/2 which uses
wgrib2's -lola on a tight bounding-box subgrid for speed (the pure
Elixir decoder takes ~70s per point on a 395 MB file; wgrib2 handles
40 points in seconds). The worker now routes through this path.
- Spike docs at docs/research/hrrr_native_levels.md confirming files
are on AWS S3 for 5+ years, 50 hybrid levels, and include TKE and
SPFH needed for Phase 2 turbulence features. Architectural finding:
per-point on-demand fetching is impractical (~530 MB/file), so
the ingestion worker batches per (date, hour) instead.
- hrrr_native_profiles schema: arrays per level plus cached surface
scalars and placeholder columns for Phase 2/4 derived fields.
Strictly additive — the existing hrrr_profiles table is untouched.
- HrrrNativeClient: pure URL/message-list helpers, build_native_profile/1
that turns a parsed wgrib2 map into the schema shape (TDD'd).
- Exposed HrrrClient.download_grib_ranges/2 so the native client
reuses the existing parallel byte-range download + disk cache.
- HrrrNativeGridWorker: Oban worker keyed on {year, month, day, hour},
unique at :infinity, pulls distinct (lat, lon) points from contacts
in the ±30 min window, downloads the native grib2, extracts per
point, bulk-upserts.
- mix hrrr_native_backfill --limit N enqueues the top-N hours by
contact count.
Phase 1 gate still pending Task 1.6 (sanity-check backtest after
live data lands).
mix backtest --feature NaiveGradient now resolves to the same function
as --feature naive_gradient. Names are normalized via Macro.underscore,
so any casing works. Also prints the available functions when an
unknown feature is requested so typos don't produce an opaque
UndefinedFunctionError deep in the Enum.map stack.
Add Microwaveprop.Backtest: a feature-evaluation framework that runs
a (lat, lon, valid_time) -> float function over the historical QSO
corpus and a matched random-time baseline, reporting distribution
statistics, distance-binned lift, and band-stratified lift.
Adds four baseline feature wrappers around the current scorer inputs
(naive_gradient, td_depression, time_of_day, pressure), a mix backtest
CLI, and the first set of baseline reports under priv/backtest_reports
so downstream phases have a frozen reference point to compare against.
Multi-phase plan to incorporate the meteorologist's feedback on
algo.md: hybrid-sigma vertical resolution, boundary-layer turbulence
features, ray-traced duct geometry, frontal geometry, radar-based
stability, climatological anomalies, regional seasonal tuning, and
5-minute METAR. Every phase has a backtest-gated stop criterion so
features that don't show lift against the historical QSO corpus
don't ship.
Wire up the Oban Pro Smart engine in base + runtime config and add the
Pro migration (oban_producers, oban_crons, oban_queues tables + the
v1.6 partition/workflow index additions on oban_jobs).
On the Smart engine the era5_batch queue now has a global_limit of 4
in-flight jobs across the cluster and a rate_limit of 10 jobs per hour
so slow Copernicus CDS submit/poll cycles can't exhaust our daily
request budget when a backfill kicks off.
Adds the commercial Oban Pro package (vendored from the towerops-web2
vendor tree) so this project can use its workers, plugins, and
Smart engine features.
The slow CDS submit/poll/download cycle of the month-batch worker was
sharing the era5 queue with the cheap Era5FetchWorker router, and its
2 concurrent slots were permanently pinned by long-running batch jobs
while hundreds of router jobs starved. Give the batch worker its own
era5_batch queue (also 2 concurrent per pod) so the router has
dedicated capacity.
Vendor oban_web 2.12.1 and oban_met 1.1.0 (taken from the
towerops-web2 vendor tree), bump oban to ~> 2.21, and mount the Oban
dashboard at /admin/oban behind the existing require_admin on_mount
hook. Adds an admin-only "Oban" nav link and extends the Dockerfile
to copy vendor/ before deps.get so production builds pick up the
path dependencies.
When a contact is linked to a logged-in user account, display that
user's callsign in the Submitted column instead of a generic "Yes".
Anonymous submissions still show "Yes"; scraped contacts show an em
dash.
Add a per-entry progress bar (with cancel) while a CSV streams in, and
replace the small post-commit alert with a prominent "X contacts
submitted" confirmation card.
Drops the formal "What we're trying to do / The approach / The stack"
scaffolding for a shorter, first-person telling: line-of-sight is
wrong, here's what we're doing about it, here's what's in the box,
here's what's next. Adds a highlighted note that the scoring is
hand-calibrated and not updated in real time, with links to the
algorithm page and the submit page so readers can help close the
loop.
Contacts now carry a user_id FK to users. When a logged-in user
submits a contact (single or CSV upload), the server injects their
user_id and the email from their account into the submission params,
and the visible email input is replaced with a hidden input holding
that value — logged-in users don't have to retype their address.
The submission changeset now requires at least one of user_id or
submitter_email (rather than hard-requiring email), and email format
is only validated when one is actually present.
Reworded the submit-page blurb from "10 GHz to 241 GHz" to "902 MHz
and up" so it matches the actual band coverage of the project.
Per-point ERA5 fetches were tragically slow because every point-hour
triggered its own asynchronous CDS job (submit → poll → assemble →
download). For backfill this meant thousands of independent jobs
queued against Copernicus. The new path groups requests by calendar
month and a 2° × 2° lat/lon tile so one CDS cycle populates ~60k
profiles at once, and Oban uniqueness on (year, month, tile_lat,
tile_lon) collapses every duplicate enqueue.
- Era5BatchClient builds the monthly CDS requests, extracts every
(lat, lon, hour) from the GRIB2 blob with wgrib2, derives
refractivity params, and bulk-inserts in 2k-row chunks with
on_conflict: :nothing. fetch_month_into_db/1 short-circuits when
the month-tile already has any cached profile.
- Era5MonthBatchWorker runs the batch on the :era5 queue with a
generous backoff (10m → 1d) and the uniqueness key above.
- Era5FetchWorker is now a thin router: cache hit → :ok, cache miss
→ enqueue the month-batch for the point's tile-month and return.
No more per-point CDS calls.
- Wgrib2 grows extract_grid_messages/3 which preserves per-message
datetimes by parsing `d=YYYYMMDDHH[MMSS]` from the inventory, so a
single GRIB2 file carrying a whole month decodes correctly.
- The era5_backfill mix task enqueues month-tile batches directly.
Uploads now show a review page with three summary cards (valid,
duplicates, invalid), tables for invalid and duplicate rows, and a
sample of the first 20 valid rows. Nothing is written to the database
until the user clicks "Looks good — submit N contacts".
Duplicate detection treats two rows as the same contact if they share
the same pair of (callsign, grid) tuples (direction-agnostic), the
same band, and timestamps within 60 minutes. Dedup runs against both
existing DB contacts (single band/time-scoped query) and earlier rows
in the same upload, and the preview labels which side matched.
CsvImport grows preview/2 + commit/1; the old import/2 is kept as a
thin backward-compat wrapper that skips dedup. Added 11 new CsvImport
tests and reworked the SubmitLive CSV tests for the two-step flow.
Pruning used to only run at the end of a successful PropagationGridWorker
pass, so a stretch of failed compute jobs (k8s OOM kills, SIGTERM)
stopped prune from running and let the table accumulate ~5h of stale
rows. A dedicated PropagationPruneWorker now runs every 15 minutes on
its own Oban cron, and PropagationGridWorker also calls prune_old_scores
at the start of each run as a second safety net. Bumped the delete
timeout from 2m to 5m so the first catch-up pass has enough headroom.
New PrivacyLive page explains that collected/submitted data is used
only for propagation research and will never be sold, traded, or
disclosed. Layouts.app now renders a small footer with the NTMS
attribution and a Privacy link on every page that uses it, so the
full-screen map pages are untouched. The duplicate NTMS credit on
the About page is removed since it now lives in the shared footer.
OSM's tile usage policy requires a Referer header identifying the site.
strict-origin-when-cross-origin is the current browser default and sends
the origin on HTTPS→HTTPS cross-origin requests; pinning it explicitly
prevents silently losing the header if a browser default ever changes.
- Beacon map defaults to just the marker at zoom 11. A daisyUI toggle
switch shows/hides the estimated-coverage cell layer; the coverage
info line and tier legend are hidden unless the toggle is on.
- Beacon detail list replaced with a compact dl grid (2-4 columns).
Lat/lon rounded to 5 decimals, power/height/beamwidth formatted via
Beacon.format_mw/1 so no more "2.5e3" scientific notation or "280.0".
format_mw/1 hoisted onto the schema so index and show share it.
- About page no longer uses prose classes (this project doesn't load
@tailwindcss/typography). Headings, lists, and paragraphs now use
explicit utility classes so they render as intended.
Beacons can now record a directional bearing ("omni" or 0-360 degrees)
and a beamwidth in degrees. The keying list grows from {on_off, fsk}
to include FM Voice, WSPR, and the full Q65A/B/C/D/E × 15/30/60/120
family, grouped via optgroups in the submission form.
- /map now centers on the visitor's Cloudflare geolocation when the
cf-iplatitude/cf-iplongitude headers are present, falling back to DFW.
Initial HRRR score bounds track the chosen center.
- Added /about link to the /map side navigation.
- Renamed "Propagation Map" label to "Propagation Prediction Map" in
the /map page title, sidebars, and the /weather back-link.
- Populated /about with mission, approach, stack, roadmap, and a live
stat grid fed from DB counts (estimates via pg_class.reltuples for
the multi-million-row tables).
- Anonymous users can submit beacons (held pending admin approval)
- Added notes field to beacons (textarea on form, shown on detail page)
- Added Beacons link to /map sidebar nav (mobile + desktop),
removed stale Rover Planner link
- Moved /backfill to /admin/backfill under the admin live_session
- Beacon detail map zooms in two extra levels after fitBounds
The Era5FetchWorker now declares an Oban unique constraint on
{lat, lon, valid_time} for any job in available/scheduled/executing/
retryable, so two backfill runs targeting the same contact grid point
can no longer spawn parallel CDS requests for the same hour.
Because Oban OSS insert_all doesn't honor unique, ERA5 jobs are now
routed through Oban.insert/1 from ContactWeatherEnqueueWorker and the
era5_backfill mix task. Other worker types still use insert_all.
