Rename the local dev database from microwaveprop_dev to prop_dev so
the name matches the production database. Add scripts/restore_prod_to_local.sh,
which pulls the latest pg_dump off the backup server, drops the local
prop_dev, recreates it, and pg_restores into it.
Add a "Forgot your password?" flow off the login page. A 24-hour
reset_password token is emailed on request, the landing page lets the
user pick a new password, and all other tokens for the user are
revoked on success.
The request endpoint returns the same flash regardless of whether the
email matches a user so that attackers can't enumerate accounts.
Adopt the handoff design system: navy-slate surfaces with indigo-violet
accent (#6366F1 dark / #5458E3 light), Inter as the UI font, hairline
1px borders, softer corner radii. Add propagation-scale and amateur-band
CSS vars so JS hooks can pull theme-aware colors via getComputedStyle.
Polish: drop table zebra striping for hover-on-base-200 rows with
uppercase small-caps headers, bump .header to text-2xl, tighten nav
spacing with a hairline divider, fix broken text-brand -> text-primary
on the auth screens.
Wire --band-* into contacts_map_hook: resolve once at mount, re-resolve
on data-theme mutation so polylines recolor when the theme toggles.
Previously the map loaded the latest cached valid_time from GridCache,
which can be any +Nh forecast hour the worker has written while walking
f00-f18. Users expect to see current conditions on load, not a forecast
that's ~11 hours out.
Docker final stage bakes BUILD_TIMESTAMP into /app/BUILD_TIMESTAMP via
build-arg (only consumed after mix compile, so earlier layer caching is
preserved). Application.build_timestamp/0 reads the file, falls back to
DEPLOY_TIMESTAMP env, then to compile time. Navbar renders a compact
"Deployed Xm ago" with the full UTC time in the tooltip.
AgentSkillsController reads priv/agent-skills/*.md at compile time via
@external_resource + File.read! to bake sha256 digests into the module.
priv/ is copied after compile to keep the compile cache stable, so add
a targeted COPY for the agent-skills subdir before compile — same
pattern as algo.md.
Implements the subset of the isitagentready.com checklist that maps to
real capabilities of this site:
- RFC 8288 Link headers on every browser response advertising
service-doc (/algo), about, privacy-policy, and sitemap.
- Markdown-for-Agents content negotiation: requests with
Accept: text/markdown for / or /algo return real markdown
(curated summary + verbatim algo.md) with x-markdown-tokens hint.
Other paths still 406 — no synthesizing markdown from LiveView HTML.
- Content-Signal directive in robots.txt declaring
search=yes, ai-train=no, ai-input=no.
- RFC 9727 API catalog at /.well-known/api-catalog with the sole
public endpoint (/api/contacts/map) linking status -> /health and
service-desc -> /openapi.json (a new minimal OpenAPI 3.0 spec).
- Agent Skills Discovery index at /.well-known/agent-skills/index.json
listing two skill documents (fetch-contacts, read-algorithm) with
sha256 digests computed at compile time; documents served at
/.well-known/agent-skills/{name}/SKILL.md.
Skipped (don't match actual site capabilities): Web Bot Auth JWKS,
OAuth/OIDC discovery, MCP server card.
Band toggles previously roundtripped through LiveView and rebuilt every
polyline + popup + hover handler on each click. Now the hook builds one
L.LayerGroup per band once after the initial fetch; toggling is just
addLayer/removeLayer on the map (O(bands), not O(contacts)). Checkbox
and All/None events are handled directly in the hook via delegated
document listeners — no LV roundtrip. Callsign filter still roundtrips
(debounced) since it's cross-cutting, and now iterates pre-built
polylines without recreating them.
The detail page was hard-coding GHz formatting, so 50/144/222/432/902
rendered as 0.1/0.4/0.9 GHz instead of the MHz labels every amateur
operator actually uses. Switch to MHz when band < 1000, keep GHz for
microwave bands.
Also surface band and mode in the header subtitle so the key QSO
metadata is visible before SRTM/terrain enrichment completes (the
elevation-profile block that previously rendered them is gated on
that enrichment).
Both workers had unique: [period: 300] which only dedupes within a
5-minute window. Re-running the enrichment-enqueue script hours or
days later stacked identical jobs in the available queue — we found
~60k redundant weather + 3k redundant hrrr jobs in prod from the
March and April enqueue passes.
:infinity means Oban dedupes against every live job in the states
list regardless of age, so a second enqueue pass is a no-op as long
as the first pass's jobs haven't completed yet.
Vendors canvas-confetti 1.9.3 and wires an ImportConfetti hook on the
alert-success banner that /imports/:id already renders when
run.status == "complete". The hook uses the "realistic look" preset
from confetti.js.org and runs on mount, so it fires exactly once —
when the banner first appears.
apply_valid_rows only matched {:ok, _} and {:error, %Changeset{}}, so when
Radio.create_contact returned {:error, :duplicate, existing} — the race
window where preview flagged a row as new but another writer inserts the
same contact before the worker picks up the chunk — the reduce callback
crashed with FunctionClauseError. Oban retried the chunk, rolling back
the Postgrex transaction and surfacing the DBConnection errors the
operator was seeing.
Treat :duplicate as a silent skip: the row is already in the DB, there
is nothing to insert, and the counter should not double-count the existing
contact as an error either.
SubmitLive now hands off to the async pipeline:
- confirm_csv/confirm_adif → CsvImport.enqueue/2 → push_navigate to
/imports/:id. ADIF preview shape matches CSV, so one enqueue call
serves both.
- allow_upload(auto_upload: false) so bytes only stream when the user
clicks submit. Progress bar + percentage now gated on
entry.progress > 0 and < 100 — invisible during file selection,
only rendered while the actual upload is in-flight.
- Old in-page csv_results / @csv_result / reset_csv removed; the
/imports/:id page replaces them.
New ImportLive at /imports/🆔 subscribes to 'csv_import:<id>' PubSub,
shows total/processed/imported/refined/error counters in a daisyUI
stats grid, progress bar, status badge, and a collapsed errors table
when error_count > 0. Missing or malformed run_id redirects to /submit
with a flash.
7 new frontend tests (5 ImportLive, 2 updated SubmitLive), 1902 total,
credo clean.
For large CSVs (e.g. the 34k-row ARRL dump), the synchronous commit
path blocked for minutes and a crash mid-commit left no audit trail.
Refactor to run inserts + enrichment enqueues via Oban.
- New import_runs table: stores preview rows + running counters (total,
processed, imported, refined, error) + status + errors map + timing.
- New :contact_import Oban queue (limit 4) + ContactImportWorker.
Args: {import_run_id, offset, limit}. Each chunk (100 rows) inserts
its slice via CsvImport.commit_rows/1, atomically increments counters
in one update_all, flips status, and broadcasts progress on
'csv_import:<id>' PubSub topic.
- CsvImport.commit_rows/1 extracted from commit/1 (back-compat preserved).
New CsvImport.enqueue/2 persists the run + dispatches N chunk jobs.
Also: submit page copy updated from '902 MHz and up' to '50 MHz and up'
matching the band allowlist expansion done earlier.
LiveView UI for /imports/:id is a separate commit.
57,186 prod contacts stored pos1/pos2 with 'lng'; 1,133 used 'lon'.
Every Elixir caller carried a `pos["lon"] || pos["lng"]` fallback
— which just caused a SQL widget to silently miscount 98% of contacts
(count_narr_done used `pos1->>'lon'` directly, no fallback, so every
lng-keyed row returned NULL and failed the coverage check).
- Migration rewrites every pos1/pos2 JSONB in place, renaming 'lng' to
'lon' and dropping 'lng'.
- Removes all 20+ `|| pos["lng"]` fallbacks across lib/, workers,
scorer, weather, radio.ex, contact show view, and recalibrator.
- lib_ml/propagation_analyze.ex SQL now reads pos1->>'lon' directly
(was reading 'lng' only, which would have broken after migration).
- priv/repo/import_contacts.exs one-time seed script now emits 'lon'
with string keys, matching production shape.
- Test fixtures in 4 test files normalized to 'lon'.
- Two lng-characterization tests deleted — nonsensical post-normalize.
- Updated notebook + old import_weather script to match.
- JS hook contact_map_hook.ts TypeScript type narrowed to 'lon'.
Visitor geolocation already flows via the Cloudflare CF-IPLatitude /
CF-IPLongitude headers into the session; MapLive centers the map on
the visitor's location (or DFW as fallback). Add a banner that appears
above the map when those coords are outside the CONUS bbox
(lat 24.5-49.5, lon -125 to -66.5), letting overseas visitors know
propagation data won't be available at their location yet.
Map still centers on the visitor's actual location — this is a note,
not a re-center. Banner is fully absent from the DOM when not needed
(:if conditional).
parse_cdo_row/1 assumed cdo -outputtab,code,lev,value would always emit
numeric codes in the first column, but Debian's cdo 2.5.1 (the version
in the prod container) emits mixed output:
- 'codeNN' with a literal 'code' prefix for most GRIB1 parameters
- Short GRIB1 names for a few: 2tag2 (PRES sfc), saip (DPT 2m),
tpag10 (HGT)
Result: every row fell through to :error, yielding a 'no parseable
values' error and 986 discarded NarrFetchWorker jobs.
Fix: normalize the code token by stripping 'code' prefix when present
and mapping known 2.5.1 short names. UGRD/VGRD intentionally omitted
(no discard evidence yet) — comment documents the add-when-they-surface
plan.
Also exposed parse_cdo_outputtab/1 as @doc false for testability.
count_narr_done/0 checked narr_profiles.valid_time within qso_timestamp
± 30 min — but NARR analyses live at 3-hour marks (00Z/03Z/.../21Z) and
NarrClient.snap_to_analysis_hour/1 floors qso_timestamp to the previous
mark before fetching. A QSO at 18:50Z snaps to 18:00Z on the fetcher
side; the profile stored at 18:00Z fell outside the widget's [18:20,
19:20] window, so the status page showed 6/208 done when actual
coverage was 208/208.
Fix: use equality against the Postgres equivalent of the snap
(date_trunc('hour', ts) - make_interval(hours => hour % 3)).
Comment cites NarrClient.snap_to_analysis_hour/1 so future changes keep
them in sync.
Backtest branch's unknown-feature guard was dead code: the if block's
{:error, _} return was discarded and execution fell through, running the
backtest with a just-created atom and writing a report for nonsense
features. Two compounding issues:
- function_exported?/3 returns false for modules not yet loaded in the
BEAM, so even valid feature names triggered the warning path on a
cold VM.
- String.to_atom/1 on caller-supplied strings created unbounded atoms.
Fix: Code.ensure_loaded?(Features) before the check, String.to_existing_atom/1
with rescue (Features exports __info__(:functions) so every valid function
atom already exists), and an explicit early return via a with pipeline so
the error tuple actually leaves the worker.
FreshnessMonitor's comment claimed 'Oban unique constraint on
PropagationGridWorker prevents duplicates' — but the worker declared
no unique: option. During a long outage the monitor's 5-min stale-check
tick would pile up identical jobs (a 2h outage = 24 stacked jobs, each
doing the full f00-f18 HRRR chain).
Put the dedup on the worker (vs. the monitor's insert call) so anything
enqueuing it benefits — FreshnessMonitor, the hourly cron, or a manual
mix propagation_grid run.
unique: [period: 3600, states: [:available, :scheduled, :executing,
:retryable]] aligns with the hourly cron cadence. The seed job args
(%{}) collapse with themselves while chain-step jobs keep distinct
run_time/forecast_hour and remain enqueuable.
build_messages_per_message/3 was slicing the wgrib2 -lola bin output as
a packed float32 stream, ignoring the 4-byte record-length header + 4-byte
trailer that Fortran unformatted records prepend/append around each
message's data block. First value of message 0 came back as ~2.24e-44
(little-endian float32 of the int-16 record length), and every subsequent
message was shifted by 8 bytes so values spilled into neighbouring cells.
Fix: propagate the correct offset math already used by parse_lola_binary/3
(record_overhead = 8, stride = bytes_per_message + 8, data_offset =
msg_idx * stride + 4).
Flipped the characterization test from metadata-only to per-cell physical
asserts (200 K < TMP, DPT < 340 K; DPT <= TMP) and a 0.01 K cross-check
against extract_grid/3 output on the same fixture + bbox. Latent bug —
no production callers of extract_grid_messages*.
16 characterization tests against real wgrib2 binary + checked-in HRRR
fixtures: extract_grid/3 (bbox + physical sanity on TMP/DPT + longitude
convention), extract_grid_from_file/3, extract_grid_from_file_mapped/4
reducer plumbing, extract_grid_messages/3 metadata, extract_points_from_file/3
nearest-neighbor snap, undefined-value sentinel filtering, empty-match and
error shapes. Tagged @describetag :slow so they run on 'mix test --include
slow' (project default excludes slow, matches existing HRRR/NARR tests).
Surfaces one real bug (not fixed): extract_grid_messages variants don't
account for the 8-byte Fortran record overhead between messages in wgrib2
-lola output, so per-message values after the first are shifted. The
correct offset math exists in parse_lola_binary/3. Tests characterize
current behavior (metadata-only); worth a follow-up fix + value-range
assertions.
10 characterization tests: fresh vs stale thresholds (>120 min),
boundary behavior, empty scores dir, supervised start, non-dedup of
repeated stale ticks, :ignore when disabled via config.
Same fix as rtma_client: MrmsClient had a hardcoded req_options/0 —
switched to a Keyword.merge with Application.get_env so tests can stub
HTTP. 13 new characterization tests: cache put/get/clear/broadcast and
worker listing/download/up-to-date paths. Full GRIB2 parse happy path
deferred to the wgrib2 coverage task.
RtmaClient was the only weather client without a Req.Test plug seam —
added the standard req_options() helper matching NarrClient/HrrrClient/
etc. so its HTTP calls can be stubbed in test.
17 characterization tests: URL construction, idx fetch 404/503, hour
truncation, malformed idx, range-GET error propagation, worker's
existing-row short-circuit, client-error propagation. GRIB2 byte
parsing paths deferred to the wgrib2 coverage task.
Status page NARR progress counted `hrrr_status = :unavailable` as the
candidate set, which swept in ~13.6K post-2014 HRRR retry rows and
showed "0 / 13,652". Candidate eligibility is qso_timestamp-based, not
hrrr_status-based — switch both the numerator and denominator to
`qso_timestamp < NarrClient.coverage_end()` so pre-2014 rows are
counted whether their hrrr_status is :queued, :unavailable, or
:complete.
ContactWeatherEnqueueWorker: short-circuit hrrr_points_for_contact/1
for pre-coverage timestamps and mark the contact :unavailable when no
HRRR jobs are built. Was producing a ping-pong where reconcile flipped
pre-2014 :queued → :unavailable and the same cron run flipped it back
to :queued from a rebuilt HRRR job that could never land data.
