Commit graph

298 commits

Author SHA1 Message Date
2f65ec8dff
Reduce HRRR worker concurrency to 3 2026-04-05 16:20:23 -05:00
94d1ce2809
Increase DB pool size to 50 to prevent connection exhaustion under load 2026-04-05 12:54:21 -05:00
8d008ba934
Add logging to WeatherFetchWorker for ASOS and RAOB fetches 2026-04-05 10:26:10 -05:00
7fd8f3165c
Add deterministic tiebreaker to contacts sort order 2026-04-05 09:27:34 -05:00
d1b1b4cc7e
Show flag emoji on contacts list for flagged_invalid contacts 2026-04-05 09:15:02 -05:00
ad1b01807c
Batch is_grid_point backfill and reduce HRRR workers to 5
- Split migration backfill into 50k-row batches to avoid container timeout
- Reduce prod HRRR worker concurrency from 10 to 5
2026-04-05 08:06:20 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00
a209fc65ae
Restrict enrichment enqueue to T-Mobile CGNAT subnet (172.56.0.0/13)
Store remote_ip in session, check subnet in LiveView mount before
enqueuing weather/HRRR/terrain/solar enrichment jobs.
2026-04-04 18:55:42 -05:00
dc66252fdc
Reduce dead tuples by using conditional upserts and skip-on-conflict
- HRRR, IEMRE, terrain: on_conflict: :nothing (immutable data)
- Surface obs, soundings, solar: conditional WHERE (only update when values differ)
- set_enrichment_status!: skip rows already at target status
2026-04-04 17:32:57 -05:00
383c18e50a
Show IEMRE gridded reanalysis data on contact detail page 2026-04-04 11:35:30 -05:00
da490f7c59
Add tooltip to enrichment badge showing per-type status 2026-04-04 11:26:35 -05:00
2a3309ca13
Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
6d2f1fb4c3
Fix swapped Submitted/Enriched column headers 2026-04-04 11:22:06 -05:00
72b272cf5c
Add enrichment status badge column to contacts table 2026-04-04 11:15:18 -05:00
915e8bdccf
Add UTC labels to Timestamp and Added column headers 2026-04-04 11:11:01 -05:00
95a2afaa01
Add inserted_at column to contacts table 2026-04-04 10:51:39 -05:00
e107706915
Show dead tuple counts on backfill dashboard, fix table visibility during vacuum
Removed n_live_tup > 0 filter that hid tables when pg_stats reports 0 rows
during VACUUM. Added dead tuple column with warning highlight to monitor
vacuum progress. Also includes Styler reformatting and unused dep cleanup.
2026-04-04 10:45:06 -05:00
23aa2786ab
Skip updating unchanged scores to reduce dead tuples, tune autovacuum
ON CONFLICT now only replaces score/factors when the score actually changed,
avoiding dead tuple generation for the ~80% of grid points that don't change
between consecutive HRRR runs.

Migration sets aggressive autovacuum on propagation_scores: zero cost delay,
2000 cost limit, 1% scale factor. The table was at 72GB with 108M dead rows
because default autovacuum couldn't keep pace with 14M upserts per hour.
2026-04-04 09:53:45 -05:00
4759c65809
Support two-callsign search on contacts page to find contacts between a pair 2026-04-04 09:29:32 -05:00
f411045242
Add layer descriptions to weather map control panel 2026-04-03 17:21:02 -05:00
bf8c6d4789
Add color scales and detail panel entries for upper-air weather layers 2026-04-03 16:54:13 -05:00
08844d3515
Add grouped upper-air and ducting layer buttons to weather map 2026-04-03 16:53:22 -05:00
9cbf8b91f4
Derive upper-air weather layers from HRRR profiles in weather grid queries 2026-04-03 16:52:01 -05:00
489a886b0e
Add WeatherLayers module for deriving upper-air map fields from HRRR profiles 2026-04-03 16:49:17 -05:00
109558f7cc
Add weather map link to propagation map sidebar 2026-04-03 15:16:05 -05:00
b176390b7b
Add WeatherMap JS hook with canvas heatmap and 6 weather layers 2026-04-03 15:15:23 -05:00
1796ee4cb7
Add WeatherMapLive LiveView with layer pill buttons 2026-04-03 15:11:55 -05:00
10390f3d07
Broadcast weather:updated PubSub after HRRR profiles stored 2026-04-03 15:10:16 -05:00
0dd695c574
Add weather grid query functions for /weather map page 2026-04-03 15:08:21 -05:00
411951147a
Use ON CONFLICT DO NOTHING for HRRR backfill to prevent table bloat 2026-04-03 13:29:15 -05:00
1c7c980cf4
Debounce backfill dashboard updates to once per second 2026-04-03 10:52:24 -05:00
8d84bd2d18
Prune propagation scores per valid_time to avoid massive single DELETE 2026-04-03 10:46:17 -05:00
33490d37d0
Chunk batch inserts to avoid Postgres parameter limit, add ANALYZE migration 2026-04-03 09:34:02 -05:00
b4012baf89
Increase Oban queue concurrency for backfill throughput 2026-04-03 09:15:04 -05:00
ebeaaa06e4
Use batch insert for HRRR profiles, tune Postgres for 16GB host 2026-04-03 09:12:24 -05:00
83ab40fdad
Add robots.txt and sitemap.xml for search engines 2026-04-03 08:44:33 -05:00
78bc577824
Show 'processing' instead of '1 job in queue' 2026-04-02 16:46:32 -05:00
f9c6e3730a
Allow retry of unavailable enrichment on page load 2026-04-02 16:46:12 -05:00
45026437da
Add Weather.sync_stations! to import ASOS/RAOB stations from IEM 2026-04-02 16:42:20 -05:00
68fdd1d2b6
Fix unique constraint blocking re-enqueues of completed jobs
- Only deduplicate against active jobs (available/scheduled/executing/
  retryable), not completed ones. Completed jobs no longer block new
  inserts for the same args.
- Period reduced from 3600s to 300s
- Fixes contacts stuck in :queued with no actual Oban jobs
2026-04-02 16:34:39 -05:00
a8920e9480
Fix position computation, grid truncation, and status resets
- Truncate odd-length grids to nearest valid even length (FN03c → FN03)
- Handle lng/lon key inconsistency in haversine distance calculation
- Reset unavailable statuses to pending when positions are computed
- Rename backfill queue column to "Pending"
2026-04-02 16:31:34 -05:00
33171ab32e
Replace polling with Postgres LISTEN/NOTIFY for live updates
- Database triggers on contacts and oban_jobs fire pg_notify on
  status/state changes
- RepoListener GenServer subscribes via Postgrex.Notifications and
  broadcasts to PubSub
- Backfill dashboard subscribes to PubSub instead of 2-second polling
- Eliminates periodic SELECT queries, updates arrive instantly
2026-04-02 16:26:22 -05:00
12d7dbf7ea
Fix phantom queued status when Oban unique constraint rejects job 2026-04-02 16:22:06 -05:00
d36256e730
Show queue position on all loading spinners, fix solar backfill
- Unified queue_info helper shows "(N jobs in queue)" on all spinners
- Solar index auto-enqueues when missing on contact page
- Freshness monitor disabled in dev
- Fix test stubs for solar client
2026-04-02 16:17:31 -05:00
0e089e8493
Auto-fetch solar index data on contact page when missing
- SolarIndexWorker accepts date arg for specific date fetch
- Contact show page enqueues solar fetch when data missing
- PubSub broadcast on solar data arrival, live-updates page
- Spinner shown while fetching
2026-04-02 16:07:38 -05:00
9fe53b4e7b
Fix terrain enqueue: check actual data not just status flag 2026-04-02 16:04:38 -05:00
bdc4d08111
Rename remaining qsos variable names to contacts in tests 2026-04-02 15:56:31 -05:00
7d13bf20ea
Fix raw SQL references to renamed contacts table 2026-04-02 15:55:15 -05:00
da75977a55
Rename qsos table to contacts, qso_id to contact_id
- Migration renames table, column, all indexes, and FK constraints
- Contact schema now references "contacts" table
- TerrainProfile foreign key renamed from qso_id to contact_id
- All code and tests updated to use contact_id
2026-04-02 15:52:19 -05:00
ae0e667cab
Fix backfill LiveStash recovery: always set all required assigns 2026-04-02 15:37:54 -05:00