Commit graph

331 commits

Author SHA1 Message Date
e99e585b66 Add type filter to backfill enqueue
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
2026-04-06 11:42:26 -05:00
FluxCD
ca6e836bd9 chore: update prop image to git.mcintire.me/graham/prop:main-1775491184-a80a981 [skip ci] 2026-04-06 16:00:21 +00:00
89ade2e8cc Bump HRRR queue concurrency from 1 to 5 per pod
15 concurrent HRRR fetches across 3 pods to process the 16,870
contacts that need fresh HRRR data from NOAA.
2026-04-06 10:54:45 -05:00
FluxCD
023fef4d8c chore: update prop image to git.mcintire.me/graham/prop:main-1775490031-fda468c [skip ci] 2026-04-06 15:41:17 +00:00
0e885001bc Widen HRRR profile spatial matching from 0.05° to 0.07°
The propagation grid stores profiles at 0.125° intervals, so contacts
up to 0.0625° from a grid point would miss with the old 0.05° tolerance.
This caused has_hrrr_profile? and find_nearest_hrrr to fail matching
against existing data, leaving contacts incorrectly stuck as pending.
2026-04-06 10:40:25 -05:00
FluxCD
a8f6a3b0e8 chore: update prop image to git.mcintire.me/graham/prop:main-1775489105-fbfd9ea [skip ci] 2026-04-06 15:26:12 +00:00
eeb92bcc19 Fix HRRR enrichment status stuck at queued/pending
enqueue_for_contact unconditionally marked contacts as :queued even
when no HRRR jobs were needed (profiles already exist). Workers never
updated contact status to :complete after storing profiles. Contacts
got stuck and were never re-processed.

- Mark :complete when no jobs generated (data already exists)
- Include :queued in backfill enrichable states for reconciliation
2026-04-06 10:24:56 -05:00
FluxCD
c9d1abafda chore: update prop image to git.mcintire.me/graham/prop:main-1775488547-a24e710 [skip ci] 2026-04-06 15:16:09 +00:00
c98aabfc15 Add BackfillEnqueueWorker tests (0→100% coverage) 2026-04-06 10:15:38 -05:00
62c9600f67 Add Tier 2 test coverage: SolarIndexWorker, HrrrFetchWorker, IemClient
SolarIndexWorker 36→100% (date-arg clause), HrrrFetchWorker 38→42%
(batch skip, backoff), IemClient 65→74% (fetch_current_asos).
2026-04-06 10:15:38 -05:00
7e52c6660d Add Tier 1 test coverage: EnrichmentStatus, Markdown, health, RemoteIp, Maidenhead
EnrichmentStatus 0→100%, HealthController 0→100%, Maidenhead 74→96%,
RemoteIp 56→94%, Markdown 0→92%. Total coverage 53→55%.
2026-04-06 10:15:38 -05:00
FluxCD
f1db86e5b1 chore: update prop image to git.mcintire.me/graham/prop:main-1775487979-6bfbf71 [skip ci] 2026-04-06 15:07:06 +00:00
08f01151ca Reduce Oban concurrency for 3-pod cluster, bump memory limit
Per-pod queues divided by 3 to maintain sane cluster totals.
Lifeline rescue reduced to 10 min to recover OOM orphans faster.
Memory limit 2Gi → 4Gi for HRRR wgrib2 processing.
2026-04-06 10:06:13 -05:00
FluxCD
63552b0cfe chore: update prop image to git.mcintire.me/graham/prop:main-1775487377-e486ec0 [skip ci] 2026-04-06 14:58:00 +00:00
091d469c6a Enable OS and Ecto data in LiveDashboard, fix progress bars
- Add ecto_psql_extras for PostgreSQL insights in dashboard
- Add os_mon for OS metrics (CPU, memory, disk)
- Fix progress bars using percentage instead of raw large numbers
2026-04-06 09:56:08 -05:00
7b0208408a Fix rolling update deadlock with pod anti-affinity
maxSurge:1 + maxUnavailable:0 deadlocks with anti-affinity since
there's no free physical host for the surge pod. Switch to
maxSurge:0 + maxUnavailable:1 to terminate one old pod first.
2026-04-06 09:52:38 -05:00
FluxCD
a743787ce8 chore: update prop image to git.mcintire.me/graham/prop:main-1775486921-784e41c [skip ci] 2026-04-06 14:49:39 +00:00
FluxCD
b749c58cc8 chore: update prop image to git.mcintire.me/graham/prop:main-1775486815-6eaa6ec [skip ci] 2026-04-06 14:48:38 +00:00
aacf51637e Add session affinity to service for stable LiveView websockets 2026-04-06 09:48:24 -05:00
2de94e2318 Add Erlang clustering via libcluster and expose LiveDashboard
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
2026-04-06 09:46:48 -05:00
FluxCD
f118452035 chore: update prop image to git.mcintire.me/graham/prop:main-1775486348-7654146 [skip ci] 2026-04-06 14:40:34 +00:00
8cca7461d4 Reduce DB pool size to 20 for 3-replica deployment
3 replicas × 50 connections exceeded max_connections during rollout.
20 per replica = 60 steady-state, 120 during rolling deploy.
2026-04-06 09:38:59 -05:00
FluxCD
cef08fe1e3 chore: update prop image to git.mcintire.me/graham/prop:main-1775486208-faa7055 [skip ci] 2026-04-06 14:37:33 +00:00
69cfda7e8c Scale to 3 replicas spread across physical hosts
- 3 replicas with pod anti-affinity on physical-host topology key
- Tolerate control-plane taint so pods can schedule on cp nodes
2026-04-06 09:36:37 -05:00
FluxCD
be0dfd898d chore: update prop image to git.mcintire.me/graham/prop:main-1775485649-949ed16 [skip ci] 2026-04-06 14:30:31 +00:00
05159c540a Trigger CI build 2026-04-06 09:27:21 -05:00
a11a3ec775 Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00
99470a6b23
Remove analyze_all_tables migration 2026-04-06 09:11:35 -05:00
5bee273442
Combine active jobs and completed cards on backfill dashboard 2026-04-06 09:03:44 -05:00
bb32e12c6a
Fix backfill progress layout - wider card, no text wrapping 2026-04-06 09:02:32 -05:00
95d7fc0c39
Show per-type enrichment progress bars on backfill dashboard 2026-04-06 08:59:38 -05:00
f4d3773e2b
Fix backfill progress to use exact counts instead of pg_stat estimates 2026-04-06 08:54:30 -05:00
0b68cb2d29
Disable propagation grid worker and add maintenance banner to map 2026-04-05 17:08:40 -05:00
2f65ec8dff
Reduce HRRR worker concurrency to 3 2026-04-05 16:20:23 -05:00
94d1ce2809
Increase DB pool size to 50 to prevent connection exhaustion under load 2026-04-05 12:54:21 -05:00
8d008ba934
Add logging to WeatherFetchWorker for ASOS and RAOB fetches 2026-04-05 10:26:10 -05:00
7fd8f3165c
Add deterministic tiebreaker to contacts sort order 2026-04-05 09:27:34 -05:00
d1b1b4cc7e
Show flag emoji on contacts list for flagged_invalid contacts 2026-04-05 09:15:02 -05:00
ad1b01807c
Batch is_grid_point backfill and reduce HRRR workers to 5
- Split migration backfill into 50k-row batches to avoid container timeout
- Reduce prod HRRR worker concurrency from 10 to 5
2026-04-05 08:06:20 -05:00
16883591b4
Database performance fixes and async backfill enqueue
- Fix score_pressure crash on nil pressure_mb (coastal HRRR points)
- Set 10-min timeout on grid score upsert transaction (was :infinity)
- Single DELETE for prune_old_scores instead of N queries in a loop
- Remove dead load_hrrr_refractivity that loaded 95k rows into nil map
- Pass selected_time to point_detail to skip latest_valid_time sub-query
- Batch station existence checks (1 query per path point, not per station)
- Batch solar index upserts via insert_all in chunks of 500
- Batch backfill_distances via single UPDATE FROM VALUES statement
- Add is_grid_point boolean + partial index to hrrr_profiles (replaces
  non-sargable modular arithmetic filter on every weather map query)
- Add partial index on contacts(qso_timestamp) WHERE pos1 IS NOT NULL
- Move backfill enqueue to Oban worker so UI returns immediately
2026-04-04 19:19:18 -05:00
a209fc65ae
Restrict enrichment enqueue to T-Mobile CGNAT subnet (172.56.0.0/13)
Store remote_ip in session, check subnet in LiveView mount before
enqueuing weather/HRRR/terrain/solar enrichment jobs.
2026-04-04 18:55:42 -05:00
dc66252fdc
Reduce dead tuples by using conditional upserts and skip-on-conflict
- HRRR, IEMRE, terrain: on_conflict: :nothing (immutable data)
- Surface obs, soundings, solar: conditional WHERE (only update when values differ)
- set_enrichment_status!: skip rows already at target status
2026-04-04 17:32:57 -05:00
383c18e50a
Show IEMRE gridded reanalysis data on contact detail page 2026-04-04 11:35:30 -05:00
da490f7c59
Add tooltip to enrichment badge showing per-type status 2026-04-04 11:26:35 -05:00
2a3309ca13
Migrate healthchecks from CHECKS to app.json 2026-04-04 11:24:56 -05:00
6d2f1fb4c3
Fix swapped Submitted/Enriched column headers 2026-04-04 11:22:06 -05:00
72b272cf5c
Add enrichment status badge column to contacts table 2026-04-04 11:15:18 -05:00
915e8bdccf
Add UTC labels to Timestamp and Added column headers 2026-04-04 11:11:01 -05:00
95a2afaa01
Add inserted_at column to contacts table 2026-04-04 10:51:39 -05:00
e107706915
Show dead tuple counts on backfill dashboard, fix table visibility during vacuum
Removed n_live_tup > 0 filter that hid tables when pg_stats reports 0 rows
during VACUUM. Added dead tuple column with warning highlight to monitor
vacuum progress. Also includes Styler reformatting and unused dep cleanup.
2026-04-04 10:45:06 -05:00