graham/prop
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
956 commits 1 branch 0 tags 8.1 MiB
Commit graph

247 commits

Author SHA1 Message Date
FluxCD
ba23ca2b3e chore: update prop image to git.mcintire.me/graham/prop:main-1776283206-6d69989 [skip ci] 2026-04-15 20:04:08 +00:00
FluxCD
fa2c9643c9 chore: update prop image to git.mcintire.me/graham/prop:main-1776282948-923c8a4 [skip ci] 2026-04-15 19:57:06 +00:00
FluxCD
6a33866133 chore: update prop image to git.mcintire.me/graham/prop:main-1776282553-83ea123 [skip ci] 2026-04-15 19:51:05 +00:00
FluxCD
33524ece3d chore: update prop image to git.mcintire.me/graham/prop:main-1776282255-f632ea8 [skip ci] 2026-04-15 19:46:04 +00:00
FluxCD
8afbe1932f chore: update prop image to git.mcintire.me/graham/prop:main-1776281892-361b9de [skip ci] 2026-04-15 19:42:03 +00:00
FluxCD
139f9f72ea chore: update prop image to git.mcintire.me/graham/prop:main-1776280977-91c8cc9 [skip ci] 2026-04-15 19:23:59 +00:00
FluxCD
1d0aa2848e chore: update prop image to git.mcintire.me/graham/prop:main-1776280381-118ad4e [skip ci] 2026-04-15 19:13:57 +00:00
FluxCD
ffc81b789e chore: update prop image to git.mcintire.me/graham/prop:main-1776279469-725a7e6 [skip ci] 2026-04-15 18:58:50 +00:00
FluxCD
cacce902d3 chore: update prop image to git.mcintire.me/graham/prop:main-1776278588-790eef1 [skip ci] 2026-04-15 18:44:46 +00:00
FluxCD
b889761a8e chore: update prop image to git.mcintire.me/graham/prop:main-1776273287-02b354f [skip ci] 2026-04-15 17:16:19 +00:00
FluxCD
fe9fb2a66f chore: update prop image to git.mcintire.me/graham/prop:main-1776272300-c3616b3 [skip ci] 2026-04-15 17:00:14 +00:00
FluxCD
84ffdcd261 chore: update prop image to git.mcintire.me/graham/prop:main-1776269371-7fcb271 [skip ci] 2026-04-15 16:11:01 +00:00
FluxCD
b76080bb0c chore: update prop image to git.mcintire.me/graham/prop:main-1776268032-e227978 [skip ci] 2026-04-15 15:48:57 +00:00
FluxCD
30b4baf792 chore: update prop image to git.mcintire.me/graham/prop:main-1776267507-222d0fd [skip ci] 2026-04-15 15:39:51 +00:00
FluxCD
c6d6d3f097 chore: update prop image to git.mcintire.me/graham/prop:main-1776265627-a9fa1d8 [skip ci] 2026-04-15 15:07:44 +00:00
FluxCD
ad31f6763d chore: update prop image to git.mcintire.me/graham/prop:main-1776265421-1a9df52 [skip ci] 2026-04-15 15:04:43 +00:00
FluxCD
4c1ef56537 chore: update prop image to git.mcintire.me/graham/prop:main-1776261922-11617e7 [skip ci] 2026-04-15 14:06:31 +00:00
FluxCD
9d2d67305e chore: update prop image to git.mcintire.me/graham/prop:main-1776260243-32fdb25 [skip ci] 2026-04-15 13:38:26 +00:00
FluxCD
3273a578b9 chore: update prop image to git.mcintire.me/graham/prop:main-1776259784-0c5cbbd [skip ci] 2026-04-15 13:35:24 +00:00
Graham McIntire
0c5cbbdc54
Bump prop pod memory limit 4Gi → 6Gi 
Even with the narrow HRRR grid pressure levels (4fa6798), pods are
still OOMKilled partway through the propagation chain. After the
HRRR fetch, the grid worker pipes 92k grid points through native
duct metric merge (~100 MB held for ~2 minutes), builds the full
weather grid cache (92k rows × ~40 fields), runs compute_scores
(460k score maps across 5 bands materialized in a list), and
writes five .ntms files — all while ERA5Submit workers keep
churning through CDS backfill in background processes.

Nodes have ~42% memory headroom on limits; the 2 Gi bump fits
comfortably. Shrinking the in-process allocations is a better
long-term fix but needs a proper pass through compute_scores and
persist_profiles. This gets prod unstuck in the meantime so the
forecast timeline can actually populate.
 2026-04-15 08:29:09 -05:00
FluxCD
38f6c37194 chore: update prop image to git.mcintire.me/graham/prop:main-1776259195-4fa6798 [skip ci] 2026-04-15 13:20:21 +00:00
FluxCD
e3fc09eee3 chore: update prop image to git.mcintire.me/graham/prop:main-1776206578-2d9e5a2 [skip ci] 2026-04-14 22:44:07 +00:00
FluxCD
1a394fa7ae chore: update prop image to git.mcintire.me/graham/prop:main-1776205868-572c285 [skip ci] 2026-04-14 22:32:04 +00:00
FluxCD
f57ea7b0c9 chore: update prop image to git.mcintire.me/graham/prop:main-1776205589-30c1018 [skip ci] 2026-04-14 22:28:03 +00:00
FluxCD
9f5b427116 chore: update prop image to git.mcintire.me/graham/prop:main-1776204813-92ef6c6 [skip ci] 2026-04-14 22:15:00 +00:00
FluxCD
7c5d187ac5 chore: update prop image to git.mcintire.me/graham/prop:main-1776204001-d9ec8a9 [skip ci] 2026-04-14 22:01:55 +00:00
FluxCD
63f2eca27f chore: update prop image to git.mcintire.me/graham/prop:main-1776203809-d5480ba [skip ci] 2026-04-14 21:58:55 +00:00
FluxCD
dc7bb5e110 chore: update prop image to git.mcintire.me/graham/prop:main-1776203569-7f1a7fb [skip ci] 2026-04-14 21:53:54 +00:00
FluxCD
28b6ca0ac4 chore: update prop image to git.mcintire.me/graham/prop:main-1776203013-e3ddf10 [skip ci] 2026-04-14 21:44:51 +00:00
FluxCD
c454fab83e chore: update prop image to git.mcintire.me/graham/prop:main-1776202569-1d99efb [skip ci] 2026-04-14 21:39:49 +00:00
FluxCD
0a8b5db6fe chore: update prop image to git.mcintire.me/graham/prop:main-1776202261-130e062 [skip ci] 2026-04-14 21:32:48 +00:00
FluxCD
2b4756e921 chore: update prop image to git.mcintire.me/graham/prop:main-1776201355-c9179b3 [skip ci] 2026-04-14 21:17:45 +00:00
FluxCD
b68dcddbc6 chore: update prop image to git.mcintire.me/graham/prop:main-1776198451-ab76d44 [skip ci] 2026-04-14 20:30:35 +00:00
FluxCD
c12868e0d3 chore: update prop image to git.mcintire.me/graham/prop:main-1776197620-07ff4f8 [skip ci] 2026-04-14 20:17:32 +00:00
FluxCD
a354bacd2d chore: update prop image to git.mcintire.me/graham/prop:main-1776194778-6c88c7e [skip ci] 2026-04-14 19:27:22 +00:00
Graham McIntire
6c88c7ec43
Merge remote-tracking branch 'origin/main' 2026-04-14 14:25:50 -05:00
Graham McIntire
33a8ba0956
Move NFS share from skippy to node3 at 10.0.15.103:/data 
node3's NVMe SSD is the new shared-storage home. Mount the whole
/data export from 10.0.15.103 at /data inside the pod (instead of
the old skippy export of /data/srtm at /srtm) so there's room for
future shared stuff — Parquet score files, maybe a DuckDB database,
or whatever the next thing is — without re-exporting a new path.

SRTM tiles now live at /data/srtm rather than /srtm. Update the
srtm_tiles_dir runtime config to match, and fix the two doc
pointers (CLAUDE.md, duckdb plan) that referenced the old path.

Already applied to the cluster and rolled out; pods confirmed
running with tiles visible at /data/srtm.
 2026-04-14 14:25:42 -05:00
FluxCD
4d33fa03e8 chore: update prop image to git.mcintire.me/graham/prop:main-1776193513-79b849e [skip ci] 2026-04-14 19:07:17 +00:00
FluxCD
94f7f09512 chore: update prop image to git.mcintire.me/graham/prop:main-1776192495-456e38d [skip ci] 2026-04-14 18:49:13 +00:00
FluxCD
325f9b769a chore: update prop image to git.mcintire.me/graham/prop:main-1776191148-6bc3e5c [skip ci] 2026-04-14 18:29:07 +00:00
FluxCD
9ddb63d99c chore: update prop image to git.mcintire.me/graham/prop:main-1776186441-7e48fbf [skip ci] 2026-04-14 17:08:22 +00:00
FluxCD
bf1a940ee5 chore: update prop image to git.mcintire.me/graham/prop:main-1776183202-434fca2 [skip ci] 2026-04-14 16:15:09 +00:00
FluxCD
89b7d39827 chore: update prop image to git.mcintire.me/graham/prop:main-1776182791-7352d06 [skip ci] 2026-04-14 16:09:08 +00:00
FluxCD
7c2ea47011 chore: update prop image to git.mcintire.me/graham/prop:main-1776181468-a950fe4 [skip ci] 2026-04-14 15:46:02 +00:00
FluxCD
800985bb33 chore: update prop image to git.mcintire.me/graham/prop:main-1776179025-a19cb04 [skip ci] 2026-04-14 15:07:46 +00:00
FluxCD
737cc14a92 chore: update prop image to git.mcintire.me/graham/prop:main-1776173499-03e7448 [skip ci] 2026-04-14 13:33:26 +00:00
FluxCD
bb6dc09b90 chore: update prop image to git.mcintire.me/graham/prop:main-1776173220-14a2284 [skip ci] 2026-04-14 13:27:24 +00:00
FluxCD
97f94e78e2 chore: update prop image to git.mcintire.me/graham/prop:main-1776172596-f85fade [skip ci] 2026-04-14 13:20:23 +00:00
FluxCD
a03178eaac chore: update prop image to git.mcintire.me/graham/prop:main-1776119477-6842a6d [skip ci] 2026-04-13 22:33:03 +00:00
FluxCD
07a8db5309 chore: update prop image to git.mcintire.me/graham/prop:main-1776118847-79e16ae [skip ci] 2026-04-13 22:22:00 +00:00