diff --git a/CLAUDE.md b/CLAUDE.md index 138cc0f9..d1eb61ef 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -230,7 +230,40 @@ User submits QSO → enqueue_for_qso() → weather/hrrr/terrain/iemre workers - Deployment name `prop` (3 replicas). Inspect with `kubectl -n prop get pods`, `kubectl -n prop logs deploy/prop`, `kubectl -n prop exec deploy/prop -- ...` - Push to `github` remote for code. The `dokku` remote is obsolete — do not push there. - Production Oban config in `config/runtime.exs` — no backfill cron, enrichment triggered by QSO submission -- Shared NFS mount `/data` in production container (server: node3 at `10.0.15.103:/data`). SRTM tiles live at `/data/srtm`. +- Shared NFS mount `/data` in production container (server: node3 at `10.0.19.103:/data`). SRTM tiles live at `/data/srtm`. + +#### NFS Storage Lifecycle & Cleanup Map + +**Directory layout on `/data`:** + +| Directory | Config Key | Content | Cleanup | +|-----------|-----------|---------|---------| +| `/data/scores/{band}/{iso}.prop` | `:propagation_scores_dir` | Per-band binary score grids (~93 KB) | ✅ `PropagationPruneWorker` (3h cutoff) + `retain_window` (48h window) | +| `/data/scores/{band}/{iso}.hrdps.prop` | same | HRDPS companion scores | ⚠️ Needs verification (HRDPS prune parity) | +| `/data/scores/profiles/{iso}.{etf,mp}.gz` | same | Full HRRR grid_data (~10 MB) | ✅ Same prune chain as scores | +| `/data/scores/weather_scalars/{iso}/{lat}_{lon}.mp.gz` | same | 5°×5° chunked weather scalars | ✅ `retain_window` deletes whole `{iso}` dirs | +| `/data/scores/weather_scalars/{iso}.hrdps/{lat}_{lon}.mp.gz` | same | HRDPS parallel scalars | ⚠️ Needs verification | +| `/data/buildings/` | `:buildings_cache_dir` | MS building footprint GeoJSON csv.gz tiles | ❌ **No cleanup** — downloaded on demand, never pruned | +| `/data/canopy/` | `:canopy_tiles_dir` | Canopy height COG tile slices | ❌ **No cleanup** — downloaded on demand, never pruned | +| `/data/srtm/` | `:srtm_tiles_dir` | SRTM elevation .hgt files | Reference data — no cleanup needed | + +**Atomic write protocol (all NFS file writes):** +1. Write to `path.tmp.` (system_time + unique integer) +2. `File.rename!(tmp, path)` (atomic on NFSv4) +3. Invalidate caches + +**Existing prune chain:** +1. `PropagationGridWorker` fires 4×/hour → seeds Rust grid chain +2. Chain completion → `retain_window` keeps only files within 48h forecast window +3. Safety net: `PropagationPruneWorker` every 15 min, deletes files older than 3h +4. `GridCachePruneWorker` every 15 min for in-memory/valkey grid cache + +**Known cleanup gaps:** +- Building footprint tiles (`/data/buildings/`) — no expiration, no cleanup +- Canopy tiles (`/data/canopy/`) — no expiration, no cleanup +- Orphaned `.tmp` files from crashed writes — no periodic sweep +- HRDPS companion files (`.hrdps.prop`, `weather_scalars/{iso}.hrdps/`) — prune parity with HRRR needs verification +- Empty parent directories after prune — no cleanup of empty dirs ### Operational gotchas - **`kubectl exec deploy/prop` may route to the backfill pod**: the `prop` and `prop-backfill` deployments share label selectors, so `deploy/prop` can attach to either. When diagnosing cron / leader-only behavior, identify the hot pod by name first: `kubectl -n prop get pods -o name | grep -E '^pod/prop-[a-z0-9]+-[a-z0-9]+$' | head -1`. The backfill pod has `PROP_ROLE=backfill` and `peer: false` on its Oban; running RPC there is misleading because plugins gated on leader (Cron, etc.) appear inert by design. diff --git a/k8s/deployment-backfill.yaml b/k8s/deployment-backfill.yaml index 3b21e215..28d9b557 100644 --- a/k8s/deployment-backfill.yaml +++ b/k8s/deployment-backfill.yaml @@ -55,7 +55,7 @@ spec: type: RuntimeDefault containers: - name: prop-backfill - image: codeberg.org/gmcintire/prop:main-1777300373-610892e + image: codeberg.org/gmcintire/prop:main-1784211861-2e72087 imagePullPolicy: IfNotPresent env: - name: POD_IP @@ -117,5 +117,5 @@ spec: volumes: - name: data nfs: - server: 10.0.15.103 + server: 10.0.19.103 path: /data diff --git a/k8s/deployment-grid-rs.yaml b/k8s/deployment-grid-rs.yaml index e51325f3..814b16d2 100644 --- a/k8s/deployment-grid-rs.yaml +++ b/k8s/deployment-grid-rs.yaml @@ -69,7 +69,7 @@ spec: # Image built by a new CI pipeline from rust/prop_grid_rs/. # Multi-arch (amd64 + arm64) — reuses the repo's existing # buildx wiring. - image: codeberg.org/gmcintire/prop-grid-rs:main-1777132414-6aa91e7 + image: codeberg.org/gmcintire/prop-grid-rs:main-1781279001-e879f29 imagePullPolicy: IfNotPresent env: # Elixir secrets bundle carries DATABASE_URL already. Rust @@ -162,5 +162,5 @@ spec: volumes: - name: data nfs: - server: 10.0.15.103 + server: 10.0.19.103 path: /data diff --git a/k8s/deployment-hrrr-point-rs.yaml b/k8s/deployment-hrrr-point-rs.yaml index 96d3cf6e..5a1eecdb 100644 --- a/k8s/deployment-hrrr-point-rs.yaml +++ b/k8s/deployment-hrrr-point-rs.yaml @@ -62,7 +62,7 @@ spec: # Shares the prop-grid-rs image so one CI pipeline produces # both binaries. k8s `command:` overrides the default # ENTRYPOINT to launch the per-QSO point worker. - image: codeberg.org/gmcintire/prop-grid-rs:main-1777132414-6aa91e7 + image: codeberg.org/gmcintire/prop-grid-rs:main-1781279001-e879f29 imagePullPolicy: IfNotPresent command: ["/usr/bin/tini", "--", "/usr/local/bin/hrrr-point-worker"] env: @@ -129,5 +129,5 @@ spec: volumes: - name: data nfs: - server: 10.0.15.103 + server: 10.0.19.103 path: /data diff --git a/k8s/deployment.yaml b/k8s/deployment.yaml index c1dddbc4..6cfc2afa 100644 --- a/k8s/deployment.yaml +++ b/k8s/deployment.yaml @@ -76,7 +76,7 @@ spec: type: RuntimeDefault containers: - name: prop - image: codeberg.org/gmcintire/prop:main-1777300373-610892e + image: codeberg.org/gmcintire/prop:main-1784211861-2e72087 imagePullPolicy: IfNotPresent env: - name: POD_IP @@ -164,5 +164,5 @@ spec: volumes: - name: data nfs: - server: 10.0.15.103 + server: 10.0.19.103 path: /data