Graham McIntire
5ec66df135
The scoring+upsert phase was ~4m40s per forecast hour and dominated wall time. Three stacked optimizations attack it from different angles. replace_scores/2 is a new hot-path writer that does DELETE WHERE valid_time = $1 followed by a plain insert_all (no ON CONFLICT resolution). The chain worker rewrites the full (valid_time, all bands) slice every forecast hour, so conflict detection was pure waste. AsosAdjustmentWorker still uses upsert_scores because it only rewrites the subset of cells near a station. factors is now nullable. Forecast hours f01-f18 pass factors: nil so the JSONB encode + toast write is skipped entirely — roughly halves the data volume per run. point_detail/4 coalesces nil to an empty map so the JS popup renders without a TypeError, and scorer_diff only pulls the most recent valid_time that still has factors (the f00 row). propagation_scores is now UNLOGGED, so inserts bypass WAL entirely. Durability tradeoff: an unclean shutdown truncates the table, but PropagationGridWorker rebuilds it from HRRR every 3h so a lost table is re-populated within one cron cycle. Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a speculative plan for a flat-file / DuckDB rewrite with explicit trigger conditions for when to pick it up (partitioning deferred too; revisit only if these three don't solve it). |
||
|---|---|---|
| .. | ||
| asos_nudge_test.exs | ||
| band_config_test.exs | ||
| duct_test.exs | ||
| grid_score_test.exs | ||
| grid_test.exs | ||
| inversion_test.exs | ||
| model_test.exs | ||
| pipeline_status_test.exs | ||
| recalibrator_test.exs | ||
| region_test.exs | ||
| score_cache_test.exs | ||
| scorer_diff_test.exs | ||
| scorer_test.exs | ||