Graham McIntire
5ec66df135
The scoring+upsert phase was ~4m40s per forecast hour and dominated wall time. Three stacked optimizations attack it from different angles. replace_scores/2 is a new hot-path writer that does DELETE WHERE valid_time = $1 followed by a plain insert_all (no ON CONFLICT resolution). The chain worker rewrites the full (valid_time, all bands) slice every forecast hour, so conflict detection was pure waste. AsosAdjustmentWorker still uses upsert_scores because it only rewrites the subset of cells near a station. factors is now nullable. Forecast hours f01-f18 pass factors: nil so the JSONB encode + toast write is skipped entirely — roughly halves the data volume per run. point_detail/4 coalesces nil to an empty map so the JS popup renders without a TypeError, and scorer_diff only pulls the most recent valid_time that still has factors (the f00 row). propagation_scores is now UNLOGGED, so inserts bypass WAL entirely. Durability tradeoff: an unclean shutdown truncates the table, but PropagationGridWorker rebuilds it from HRRR every 3h so a lost table is re-populated within one cron cycle. Also adds docs/plans/2026-04-14-duckdb-scores-storage.md — a speculative plan for a flat-file / DuckDB rewrite with explicit trigger conditions for when to pick it up (partitioning deferred too; revisit only if these three don't solve it). |
||
|---|---|---|
| .. | ||
| algo-reports | ||
| plans | ||
| research | ||
| algo_commercial.md | ||
| algo_original.md | ||
| algo_updated_with_data.md | ||
| analysis_output.txt | ||
| analysis_output_solar.txt | ||
| findings10.md | ||
| goes-r-research.md | ||
| RRFS.md | ||