graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
1704 commits 1 branch 0 tags 8.1 MiB
Commit graph

1704 commits

This branch
This branch
All branches
Author SHA1 Message Date
FluxCD
5583fb6167 chore: update prop image to git.mcintire.me/graham/prop:main-1775512945-6ab4ba7 [skip ci] 2026-04-06 17:04:37 -05:00
Graham McIntire
d63653e9a1 Bump terrain queue concurrency to 3 per pod 2026-04-06 17:04:26 -05:00
FluxCD
7c7dc80acf chore: update prop image to git.mcintire.me/graham/prop:main-1775512945-6ab4ba7 [skip ci] 2026-04-06 22:03:25 +00:00
Graham McIntire
f2e6512974 Re-enable hourly propagation grid scoring in production 2026-04-06 17:02:18 -05:00
Graham McIntire
c68e1ba108 Re-enable hourly propagation grid scoring in production 2026-04-06 17:02:12 -05:00
FluxCD
da4979751d chore: update prop image to git.mcintire.me/graham/prop:main-1775512477-be9b890 [skip ci] 2026-04-06 17:02:12 -05:00
FluxCD
25df2e6766 chore: update prop image to git.mcintire.me/graham/prop:main-1775509847-dcf5190 [skip ci] 2026-04-06 17:02:12 -05:00
FluxCD
c6c4214a91 chore: update prop image to git.mcintire.me/graham/prop:main-1775512477-be9b890 [skip ci] 2026-04-06 21:56:05 +00:00
Graham McIntire
79130a39e8 Mark terrain status complete when profile already exists 
TerrainProfileWorker early-exited with :ok when a profile existed
but never updated the contact status, leaving contacts stuck at queued.
 2026-04-06 16:54:29 -05:00
Graham McIntire
25d7ec317a Mark terrain status complete when profile already exists 
TerrainProfileWorker early-exited with :ok when a profile existed
but never updated the contact status, leaving contacts stuck at queued.
 2026-04-06 16:54:23 -05:00
FluxCD
00bacae2c2 chore: update prop image to git.mcintire.me/graham/prop:main-1775499376-15ec8af [skip ci] 2026-04-06 16:54:23 -05:00
FluxCD
366258e329 chore: update prop image to git.mcintire.me/graham/prop:main-1775509847-dcf5190 [skip ci] 2026-04-06 21:11:31 +00:00
Graham McIntire
646467cf4d Store stub records on empty weather fetches to prevent infinite backfill retries 
When ASOS/RAOB/IEMRE APIs return empty data, store a stub record so
dedup checks (has_surface_observations?, has_sounding?, has_iemre_observation?)
see coverage and don't recreate the same jobs on future backfill runs.
 2026-04-06 16:10:39 -05:00
Graham McIntire
1f7e9e18a4 Store stub records on empty weather fetches to prevent infinite backfill retries 
When ASOS/RAOB/IEMRE APIs return empty data, store a stub record so
dedup checks (has_surface_observations?, has_sounding?, has_iemre_observation?)
see coverage and don't recreate the same jobs on future backfill runs.
 2026-04-06 16:10:28 -05:00
FluxCD
a6ae6ce973 chore: update prop image to git.mcintire.me/graham/prop:main-1775499376-15ec8af [skip ci] 2026-04-06 18:17:26 +00:00
Graham McIntire
c3819112fe Persist checkbox and limit state across backfill enqueue submissions 2026-04-06 12:46:08 -05:00
FluxCD
ef44109351 chore: update prop image to git.mcintire.me/graham/prop:main-1775494893-2fd0175 [skip ci] 2026-04-06 17:02:37 +00:00
Graham McIntire
98289ec16e Fix format_number reversing digits 
Enum.reverse/1 was reversing each chunk's characters, scrambling the
displayed numbers (e.g. 58,363 displayed as 85,363).
 2026-04-06 12:01:25 -05:00
FluxCD
f9e2ba294d chore: update prop image to git.mcintire.me/graham/prop:main-1775494509-975e6c9 [skip ci] 2026-04-06 16:56:34 +00:00
Graham McIntire
9ffeba3fc9 Suppress health check request logs 
/health gets hit constantly by k8s probes, drowning out useful logs.
 2026-04-06 11:55:01 -05:00
FluxCD
88783765f1 chore: update prop image to git.mcintire.me/graham/prop:main-1775494141-8768175 [skip ci] 2026-04-06 16:50:32 +00:00
Graham McIntire
4ba7269e57 Prioritize pending contacts over queued in backfill 
Queued contacts just need reconciliation (check if data exists),
while pending contacts need actual HRRR fetches. Process pending
first so real work happens before reconciliation.
 2026-04-06 11:48:55 -05:00
FluxCD
89eafcb1b0 chore: update prop image to git.mcintire.me/graham/prop:main-1775493753-3737842 [skip ci] 2026-04-06 16:43:31 +00:00
Graham McIntire
e99e585b66 Add type filter to backfill enqueue 
Backfill dashboard now has checkboxes to select which enrichment types
to process (HRRR, Weather, Terrain, IEMRE). Prevents weather jobs from
flooding the queue when only HRRR needs work.
 2026-04-06 11:42:26 -05:00
FluxCD
ca6e836bd9 chore: update prop image to git.mcintire.me/graham/prop:main-1775491184-a80a981 [skip ci] 2026-04-06 16:00:21 +00:00
Graham McIntire
89ade2e8cc Bump HRRR queue concurrency from 1 to 5 per pod 
15 concurrent HRRR fetches across 3 pods to process the 16,870
contacts that need fresh HRRR data from NOAA.
 2026-04-06 10:54:45 -05:00
FluxCD
023fef4d8c chore: update prop image to git.mcintire.me/graham/prop:main-1775490031-fda468c [skip ci] 2026-04-06 15:41:17 +00:00
Graham McIntire
0e885001bc Widen HRRR profile spatial matching from 0.05° to 0.07° 
The propagation grid stores profiles at 0.125° intervals, so contacts
up to 0.0625° from a grid point would miss with the old 0.05° tolerance.
This caused has_hrrr_profile? and find_nearest_hrrr to fail matching
against existing data, leaving contacts incorrectly stuck as pending.
 2026-04-06 10:40:25 -05:00
FluxCD
a8f6a3b0e8 chore: update prop image to git.mcintire.me/graham/prop:main-1775489105-fbfd9ea [skip ci] 2026-04-06 15:26:12 +00:00
Graham McIntire
eeb92bcc19 Fix HRRR enrichment status stuck at queued/pending 
enqueue_for_contact unconditionally marked contacts as :queued even
when no HRRR jobs were needed (profiles already exist). Workers never
updated contact status to :complete after storing profiles. Contacts
got stuck and were never re-processed.

- Mark :complete when no jobs generated (data already exists)
- Include :queued in backfill enrichable states for reconciliation
 2026-04-06 10:24:56 -05:00
FluxCD
c9d1abafda chore: update prop image to git.mcintire.me/graham/prop:main-1775488547-a24e710 [skip ci] 2026-04-06 15:16:09 +00:00
Graham McIntire
c98aabfc15 Add BackfillEnqueueWorker tests (0→100% coverage) 2026-04-06 10:15:38 -05:00
Graham McIntire
62c9600f67 Add Tier 2 test coverage: SolarIndexWorker, HrrrFetchWorker, IemClient 
SolarIndexWorker 36→100% (date-arg clause), HrrrFetchWorker 38→42%
(batch skip, backoff), IemClient 65→74% (fetch_current_asos).
 2026-04-06 10:15:38 -05:00
Graham McIntire
7e52c6660d Add Tier 1 test coverage: EnrichmentStatus, Markdown, health, RemoteIp, Maidenhead 
EnrichmentStatus 0→100%, HealthController 0→100%, Maidenhead 74→96%,
RemoteIp 56→94%, Markdown 0→92%. Total coverage 53→55%.
 2026-04-06 10:15:38 -05:00
FluxCD
f1db86e5b1 chore: update prop image to git.mcintire.me/graham/prop:main-1775487979-6bfbf71 [skip ci] 2026-04-06 15:07:06 +00:00
Graham McIntire
08f01151ca Reduce Oban concurrency for 3-pod cluster, bump memory limit 
Per-pod queues divided by 3 to maintain sane cluster totals.
Lifeline rescue reduced to 10 min to recover OOM orphans faster.
Memory limit 2Gi → 4Gi for HRRR wgrib2 processing.
 2026-04-06 10:06:13 -05:00
FluxCD
63552b0cfe chore: update prop image to git.mcintire.me/graham/prop:main-1775487377-e486ec0 [skip ci] 2026-04-06 14:58:00 +00:00
Graham McIntire
091d469c6a Enable OS and Ecto data in LiveDashboard, fix progress bars 
- Add ecto_psql_extras for PostgreSQL insights in dashboard
- Add os_mon for OS metrics (CPU, memory, disk)
- Fix progress bars using percentage instead of raw large numbers
 2026-04-06 09:56:08 -05:00
Graham McIntire
7b0208408a Fix rolling update deadlock with pod anti-affinity 
maxSurge:1 + maxUnavailable:0 deadlocks with anti-affinity since
there's no free physical host for the surge pod. Switch to
maxSurge:0 + maxUnavailable:1 to terminate one old pod first.
 2026-04-06 09:52:38 -05:00
FluxCD
a743787ce8 chore: update prop image to git.mcintire.me/graham/prop:main-1775486921-784e41c [skip ci] 2026-04-06 14:49:39 +00:00
FluxCD
b749c58cc8 chore: update prop image to git.mcintire.me/graham/prop:main-1775486815-6eaa6ec [skip ci] 2026-04-06 14:48:38 +00:00
Graham McIntire
aacf51637e Add session affinity to service for stable LiveView websockets 2026-04-06 09:48:24 -05:00
Graham McIntire
2de94e2318 Add Erlang clustering via libcluster and expose LiveDashboard 
- libcluster with Kubernetes IP strategy for pod discovery
- RBAC for pod list/get, POD_IP from downward API, shared cookie
- LiveDashboard at /dashboard for all environments
 2026-04-06 09:46:48 -05:00
FluxCD
f118452035 chore: update prop image to git.mcintire.me/graham/prop:main-1775486348-7654146 [skip ci] 2026-04-06 14:40:34 +00:00
Graham McIntire
8cca7461d4 Reduce DB pool size to 20 for 3-replica deployment 
3 replicas × 50 connections exceeded max_connections during rollout.
20 per replica = 60 steady-state, 120 during rolling deploy.
 2026-04-06 09:38:59 -05:00
FluxCD
cef08fe1e3 chore: update prop image to git.mcintire.me/graham/prop:main-1775486208-faa7055 [skip ci] 2026-04-06 14:37:33 +00:00
Graham McIntire
69cfda7e8c Scale to 3 replicas spread across physical hosts 
- 3 replicas with pod anti-affinity on physical-host topology key
- Tolerate control-plane taint so pods can schedule on cp nodes
 2026-04-06 09:36:37 -05:00
FluxCD
be0dfd898d chore: update prop image to git.mcintire.me/graham/prop:main-1775485649-949ed16 [skip ci] 2026-04-06 14:30:31 +00:00
Graham McIntire
05159c540a Trigger CI build 2026-04-06 09:27:21 -05:00
Graham McIntire
a11a3ec775 Add k8s manifests and Forgejo CI for cluster deployment 2026-04-06 09:25:24 -05:00