Commit graph

271 commits

Author SHA1 Message Date
10390f3d07
Broadcast weather:updated PubSub after HRRR profiles stored 2026-04-03 15:10:16 -05:00
0dd695c574
Add weather grid query functions for /weather map page 2026-04-03 15:08:21 -05:00
411951147a
Use ON CONFLICT DO NOTHING for HRRR backfill to prevent table bloat 2026-04-03 13:29:15 -05:00
1c7c980cf4
Debounce backfill dashboard updates to once per second 2026-04-03 10:52:24 -05:00
8d84bd2d18
Prune propagation scores per valid_time to avoid massive single DELETE 2026-04-03 10:46:17 -05:00
33490d37d0
Chunk batch inserts to avoid Postgres parameter limit, add ANALYZE migration 2026-04-03 09:34:02 -05:00
b4012baf89
Increase Oban queue concurrency for backfill throughput 2026-04-03 09:15:04 -05:00
ebeaaa06e4
Use batch insert for HRRR profiles, tune Postgres for 16GB host 2026-04-03 09:12:24 -05:00
83ab40fdad
Add robots.txt and sitemap.xml for search engines 2026-04-03 08:44:33 -05:00
78bc577824
Show 'processing' instead of '1 job in queue' 2026-04-02 16:46:32 -05:00
f9c6e3730a
Allow retry of unavailable enrichment on page load 2026-04-02 16:46:12 -05:00
45026437da
Add Weather.sync_stations! to import ASOS/RAOB stations from IEM 2026-04-02 16:42:20 -05:00
68fdd1d2b6
Fix unique constraint blocking re-enqueues of completed jobs
- Only deduplicate against active jobs (available/scheduled/executing/
  retryable), not completed ones. Completed jobs no longer block new
  inserts for the same args.
- Period reduced from 3600s to 300s
- Fixes contacts stuck in :queued with no actual Oban jobs
2026-04-02 16:34:39 -05:00
a8920e9480
Fix position computation, grid truncation, and status resets
- Truncate odd-length grids to nearest valid even length (FN03c → FN03)
- Handle lng/lon key inconsistency in haversine distance calculation
- Reset unavailable statuses to pending when positions are computed
- Rename backfill queue column to "Pending"
2026-04-02 16:31:34 -05:00
33171ab32e
Replace polling with Postgres LISTEN/NOTIFY for live updates
- Database triggers on contacts and oban_jobs fire pg_notify on
  status/state changes
- RepoListener GenServer subscribes via Postgrex.Notifications and
  broadcasts to PubSub
- Backfill dashboard subscribes to PubSub instead of 2-second polling
- Eliminates periodic SELECT queries, updates arrive instantly
2026-04-02 16:26:22 -05:00
12d7dbf7ea
Fix phantom queued status when Oban unique constraint rejects job 2026-04-02 16:22:06 -05:00
d36256e730
Show queue position on all loading spinners, fix solar backfill
- Unified queue_info helper shows "(N jobs in queue)" on all spinners
- Solar index auto-enqueues when missing on contact page
- Freshness monitor disabled in dev
- Fix test stubs for solar client
2026-04-02 16:17:31 -05:00
0e089e8493
Auto-fetch solar index data on contact page when missing
- SolarIndexWorker accepts date arg for specific date fetch
- Contact show page enqueues solar fetch when data missing
- PubSub broadcast on solar data arrival, live-updates page
- Spinner shown while fetching
2026-04-02 16:07:38 -05:00
9fe53b4e7b
Fix terrain enqueue: check actual data not just status flag 2026-04-02 16:04:38 -05:00
bdc4d08111
Rename remaining qsos variable names to contacts in tests 2026-04-02 15:56:31 -05:00
7d13bf20ea
Fix raw SQL references to renamed contacts table 2026-04-02 15:55:15 -05:00
da75977a55
Rename qsos table to contacts, qso_id to contact_id
- Migration renames table, column, all indexes, and FK constraints
- Contact schema now references "contacts" table
- TerrainProfile foreign key renamed from qso_id to contact_id
- All code and tests updated to use contact_id
2026-04-02 15:52:19 -05:00
ae0e667cab
Fix backfill LiveStash recovery: always set all required assigns 2026-04-02 15:37:54 -05:00
d275e9e7c4
fix enrichment 2026-04-02 15:30:41 -05:00
d95098daa9
Remove queue pausing from PropagationGridWorker 2026-04-02 15:30:06 -05:00
15a09f0d7b
Fix enrichment: check actual data presence, not just status flag 2026-04-02 15:27:18 -05:00
f897a9a321
Re-enqueue enrichment jobs when status is queued but data missing
Previously, if a job was cancelled or lost while status was :queued,
viewing the contact detail page would not re-enqueue. Now HRRR,
weather/soundings, and terrain all re-enqueue when :queued with no
data, relying on Oban uniqueness for deduplication.
2026-04-02 15:07:49 -05:00
1fa8681c54
Keep hrrr queue running during propagation grid worker
Don't pause the hrrr queue while PropagationGridWorker runs,
so on-demand HRRR fetches from contact detail pages can proceed.
2026-04-02 15:02:23 -05:00
bb1c46fd57
Fix backfill page 500 by using pg_stat estimates instead of count(*)
The 6 sequential count(*) queries on large tables were timing out
at 15s when concurrent HRRR batch inserts held locks. Reuse the
n_live_tup estimates already fetched from pg_stat_user_tables.
2026-04-02 14:55:53 -05:00
66de041886
Show HRRR queue position on contact detail page
When HRRR data is loading, display how many jobs are ahead
in the queue so users can estimate wait time.
2026-04-02 14:42:45 -05:00
1c2879c466
Fix ambiguous column reference in backfill DB stats query 2026-04-02 13:29:15 -05:00
faa23a9693
Add Data Sources section to contact detail page
Shows provenance of all data used in the analysis:
- HRRR: path profile count, pressure levels, valid time, grid point
- Elevation: sample count, path distance, source (SRTM), duct layers
- Terrain: sample count, verdict, max elevation, diffraction loss
- Surface observations: station count and time range
- Soundings: profile count, stations, levels, ducting detection
- Solar indices: SFI, SSN, Ap
2026-04-02 13:28:58 -05:00
2452f1f397
Add detailed database stats to backfill dashboard
- Enrichment status breakdown per type (HRRR/Weather/Terrain/IEMRE)
  with color-coded status counts (complete/pending/queued/failed)
- Database overview: total DB size, HRRR profiles, observations,
  soundings with per-contact averages
- Table sizes: rows and disk size per table, sorted by size
- All stats auto-refresh every 2 seconds
2026-04-02 13:24:43 -05:00
59665a5009
Add LiveStash to backfill page, recover limit on reconnect 2026-04-02 13:13:23 -05:00
c1da0f20b1
Add LiveStash for state recovery across reconnects
- Installed live_stash ~> 0.1 with browser memory adapter
- Map page: recovers selected_band and selected_time on reconnect
- Submit page: recovers active_tab (single/csv) on reconnect
- JS params initialized with initLiveStash wrapper
2026-04-02 13:12:17 -05:00
2f9d645c50
Configure outbound SMTP email via Swoosh with TLS 2026-04-02 13:08:50 -05:00
2f65c4648c
Add /health endpoint and CHECKS file for zero-downtime deploys 2026-04-02 13:01:42 -05:00
20ed47397b
Improve map page mobile usability
Collapsible control panel with hamburger toggle, detail panel as
bottom sheet, compact legend moved to top-right, and scaled timeline
buttons for narrow viewports.
2026-04-02 12:59:55 -05:00
dc0121a520
Remove noisy SRTM tiles log 2026-04-02 12:53:46 -05:00
f542c96e46
Remove unused alias in migration 2026-04-02 12:40:14 -05:00
1da1cbed43
Fix nil access bugs found in security/bug audit
- TerrainProfileWorker: guard pos1/pos2 nil before accessing lat/lon,
  mark unavailable if missing instead of crashing
- Propagation analysis: guard pos1 nil in compute_factors
- contact_path_points: return [] instead of [{nil, nil}] on missing coords
- JS elevation hook: null check canvas before getContext
2026-04-02 12:38:49 -05:00
1cf92b2166
Cap propagation reach flood fill to 300 km radius
Without a distance limit, the BFS would expand across the entire grid
when conditions were broadly favorable, producing a polygon covering
most of CONUS.
2026-04-02 12:38:07 -05:00
2b14e836f1
Fix CSV parsing and page number input validation
- RFC 4180 CSV parser: handles quoted fields with commas and escaped
  quotes ("Smith, John" no longer breaks parsing)
- Safe page number parsing: non-numeric input defaults to page 1
  instead of raising ArgumentError
2026-04-02 12:32:59 -05:00
79af4e4959
Ensure positions computed from grids before any enrichment
- Radio.ensure_positions!/1 computes pos1/pos2/distance from grids
- Called in contact show page mount and enqueue_for_contact
- Migration backfills positions for existing contacts missing them
- Migration corrects enrichment statuses based on actual data presence
- Backfill dashboard includes contacts with grids but no positions
2026-04-02 12:28:48 -05:00
df27209025
Fix migrations: run after Repo starts, not before 2026-04-02 12:19:02 -05:00
93f2c32971
Remove timex, earmark, dns_cluster deps; add local implementations
- Replace Timex with regex-based timestamp parser (broader format support)
- Replace Earmark with local Markdown.to_html! (headings, code blocks,
  tables, lists, inline formatting, links, horizontal rules)
- Remove dns_cluster (unused, single-instance deployment)
- Add stream_data property tests for timestamp parsing (5 properties
  covering ISO, US dates, AM/PM, format roundtrips, garbage rejection)
- Removes transitive deps: combine, tzdata, hackney, certifi, metrics,
  mimerl, parse_trans, ssl_verify_fun
2026-04-02 12:17:37 -05:00
55a51f69ab
Replace boolean enrichment flags with enum status fields
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
  with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
  submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
2026-04-02 12:07:51 -05:00
fa5fabe02e
Fix stat card dotted border, use job queue counts for progress 2026-04-02 11:55:01 -05:00
4999548a0c
Show actual missing data counts instead of queued flags on backfill page 2026-04-02 11:50:47 -05:00
9485df4b8d
Read Cf-Connecting-Ip header for real client IP through Cloudflare tunnel 2026-04-02 11:49:11 -05:00