Commit graph

22 commits

Author SHA1 Message Date
5bee273442
Combine active jobs and completed cards on backfill dashboard 2026-04-06 09:03:44 -05:00
bb32e12c6a
Fix backfill progress layout - wider card, no text wrapping 2026-04-06 09:02:32 -05:00
95d7fc0c39
Show per-type enrichment progress bars on backfill dashboard 2026-04-06 08:59:38 -05:00
f4d3773e2b
Fix backfill progress to use exact counts instead of pg_stat estimates 2026-04-06 08:54:30 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00
e107706915
Show dead tuple counts on backfill dashboard, fix table visibility during vacuum
Removed n_live_tup > 0 filter that hid tables when pg_stats reports 0 rows
during VACUUM. Added dead tuple column with warning highlight to monitor
vacuum progress. Also includes Styler reformatting and unused dep cleanup.
2026-04-04 10:45:06 -05:00
1c7c980cf4
Debounce backfill dashboard updates to once per second 2026-04-03 10:52:24 -05:00
a8920e9480
Fix position computation, grid truncation, and status resets
- Truncate odd-length grids to nearest valid even length (FN03c → FN03)
- Handle lng/lon key inconsistency in haversine distance calculation
- Reset unavailable statuses to pending when positions are computed
- Rename backfill queue column to "Pending"
2026-04-02 16:31:34 -05:00
33171ab32e
Replace polling with Postgres LISTEN/NOTIFY for live updates
- Database triggers on contacts and oban_jobs fire pg_notify on
  status/state changes
- RepoListener GenServer subscribes via Postgrex.Notifications and
  broadcasts to PubSub
- Backfill dashboard subscribes to PubSub instead of 2-second polling
- Eliminates periodic SELECT queries, updates arrive instantly
2026-04-02 16:26:22 -05:00
7d13bf20ea
Fix raw SQL references to renamed contacts table 2026-04-02 15:55:15 -05:00
da75977a55
Rename qsos table to contacts, qso_id to contact_id
- Migration renames table, column, all indexes, and FK constraints
- Contact schema now references "contacts" table
- TerrainProfile foreign key renamed from qso_id to contact_id
- All code and tests updated to use contact_id
2026-04-02 15:52:19 -05:00
ae0e667cab
Fix backfill LiveStash recovery: always set all required assigns 2026-04-02 15:37:54 -05:00
bb1c46fd57
Fix backfill page 500 by using pg_stat estimates instead of count(*)
The 6 sequential count(*) queries on large tables were timing out
at 15s when concurrent HRRR batch inserts held locks. Reuse the
n_live_tup estimates already fetched from pg_stat_user_tables.
2026-04-02 14:55:53 -05:00
1c2879c466
Fix ambiguous column reference in backfill DB stats query 2026-04-02 13:29:15 -05:00
2452f1f397
Add detailed database stats to backfill dashboard
- Enrichment status breakdown per type (HRRR/Weather/Terrain/IEMRE)
  with color-coded status counts (complete/pending/queued/failed)
- Database overview: total DB size, HRRR profiles, observations,
  soundings with per-contact averages
- Table sizes: rows and disk size per table, sorted by size
- All stats auto-refresh every 2 seconds
2026-04-02 13:24:43 -05:00
59665a5009
Add LiveStash to backfill page, recover limit on reconnect 2026-04-02 13:13:23 -05:00
79af4e4959
Ensure positions computed from grids before any enrichment
- Radio.ensure_positions!/1 computes pos1/pos2/distance from grids
- Called in contact show page mount and enqueue_for_contact
- Migration backfills positions for existing contacts missing them
- Migration corrects enrichment statuses based on actual data presence
- Backfill dashboard includes contacts with grids but no positions
2026-04-02 12:28:48 -05:00
55a51f69ab
Replace boolean enrichment flags with enum status fields
- New fields: hrrr_status, weather_status, terrain_status, iemre_status
  with values: pending, queued, processing, complete, failed, unavailable
- EnrichmentStatus module defines valid state transitions
- Migration backfills: queued=true → complete, false → pending
- Workers set status on transitions (queued on enqueue, complete on finish)
- Show page reads status from contact struct, not computed dynamically
- Backfill dashboard queries status fields for accurate progress counts
- Remove cron-scheduled ContactWeatherEnqueueWorker (enrichment only on
  submission, page view, or manual backfill)
- Partial indexes on status != 'complete' for fast unprocessed lookups
2026-04-02 12:07:51 -05:00
fa5fabe02e
Fix stat card dotted border, use job queue counts for progress 2026-04-02 11:55:01 -05:00
4999548a0c
Show actual missing data counts instead of queued flags on backfill page 2026-04-02 11:50:47 -05:00
b6a5e709b7
Show enrichment progress bar with remaining/total contacts 2026-04-02 11:47:20 -05:00
c95da8ccdb
Add /backfill dashboard for enqueuing and monitoring enrichment jobs
- Input field to enqueue N contacts for backfill
- Live stats: unprocessed contacts, active/queued jobs, completed/hour
- Per-worker breakdown: executing (with spinner), available, retryable
- Auto-refreshes every 2 seconds
2026-04-02 11:46:49 -05:00