Commit graph

2 commits

Author SHA1 Message Date
7ee19c5b6a
perf+test: cache nearest_hrrr per cell, batch pg_inherits, +property tests
Two performance fixes the property tests pinned down:

* CalibrationSampler.build_for_hour/1 walked every cell through
  nearest_hrrr/3 twice — once in the producer and again in
  build_sample. Now computed once into a (band, lat, lon) → hrrr_or_nil
  map, halving the O(cells × profiles) scan (~10M comparisons saved
  per fire at typical sizes).
* PartitionManager.ensure_quarterly_partitions/2 issued one
  pg_inherits scan per (parent × lookahead). Refactored to one scan
  per parent with in-memory coverage check across all quarters.

Property tests for both modules:

* PartitionManager: tile-coverage invariant (no gaps, no overlaps),
  exact 3-month bounds, name format, multi-call idempotence,
  multi-parent independence — caught a real NaiveDateTime sort bug
  in the original test helpers. Default Enum.sort_by/2 falls back to
  term comparison on NaiveDateTime; now uses NaiveDateTime as the
  comparator module.
* CalibrationSampler: enqueued points = unique missing midpoints,
  empty enqueue when fully covered, sample row count invariant under
  HRRR availability.

Saved operational gotchas to CLAUDE.md (Oban leader election spans
all app=prop pods, kubectl exec deploy/prop ambiguity, half-year
partition coexistence, HRRR f000 publish lag).

3310 tests + 228 properties, 0 failures.
2026-05-05 17:40:13 -05:00
413cc92cab
feat(infra): auto-extend quarterly partitions for hrrr/hrdps tables
Partition list for hrrr_profiles + hrdps_profiles was hand-edited in
priv/repo/structure.sql; if no human extended it, writes for the next
quarter would silently fail at the worker level (Rust hrrr-point-worker
marks tasks failed; the calibration pipeline silently joins NULLs).

Adds:

* Microwaveprop.PartitionManager — runtime DDL helper. Quarter math via
  a single integer index (year*4 + quarter) for rollover safety.
  Coverage check via pg_inherits before CREATE so a quarterly target
  inside an existing wider partition (some 2027 ones are half-year)
  is treated as :exists rather than colliding.
* Microwaveprop.Workers.PartitionMaintenanceWorker — daily cron at
  03:15 UTC, 4-quarter lookahead. Idempotent; cheap when nothing's
  missing. lookback_quarters arg lets ops widen for catch-up.

Tests cover the create + idempotent re-run + name format + year
rollover paths via a synthetic parent created inside the sandbox
transaction (no DDL leaks into the shared test DB).
2026-05-05 16:22:16 -05:00