perf(db): tune autovacuum per-table + one-shot ANALYZE helper

Two issues spotted in pg_stat_user_tables:

1. Most `hrrr_profiles` partitions have `last_autoanalyze IS NULL`.
   The partitions are append-only from the hourly grid worker, so
   autovacuum never crosses the analyze threshold — but without
   stats the planner falls back to estimates and picks bad joins
   on per-QSO lookups.

2. `contacts`, `grid_tasks`, `hrrr_fetch_tasks`, and
   `contact_common_volume_radar` see high UPDATE volumes
   (enrichment status flips, queue claim/complete cycles, mechanism
   classifier writes) but only accrue ~5% dead before the default
   20% / 10% autovacuum triggers fire. BackfillEnqueueWorker's
   status-priority ORDER BY + the Rust workers' FOR UPDATE SKIP
   LOCKED claim scan both want fresher stats than that.

Changes:
- Migration `20260424204656_tune_autovacuum_for_high_churn` sets
  per-table `autovacuum_*_scale_factor` + `_threshold = 50` on the
  four tables above. Aggressive enough to keep stats current;
  tame enough that vacuum doesn't thrash the Turing Pi 2 Postgres
  node. Reversible.
- `Weather.analyze_all/1` walks `pg_stat_user_tables`, runs
  `ANALYZE` on every table not auto-analyzed in the last 6 h,
  reports counts. Meant to be invoked once via
  `bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'`
  after deploy to seed statistics on the cold partitions;
  idempotent on re-run.
This commit is contained in:
Graham McIntire 2026-04-24 15:48:42 -05:00
parent 609f04f649
commit 03c91f4098
No known key found for this signature in database
GPG key ID: F4ABF488E6029E59
2 changed files with 119 additions and 0 deletions

View file

@ -410,6 +410,59 @@ defmodule Microwaveprop.Weather do
{:ok, n}
end
@doc """
Run `ANALYZE` on every public-schema table that has not been auto-
analyzed recently. Callable from a release shell:
bin/microwaveprop rpc 'Microwaveprop.Weather.analyze_all()'
Surfaces the HRRR partition problem: most partitions have zero
analyze stats, so the planner picks nested-loop joins that scan
81 M rows. Running ANALYZE once gives it live_tup and column
histograms to work with.
`skip_recent` skips anything auto-analyzed in the last N seconds
(default 6 h) so re-running is cheap.
"""
@spec analyze_all(keyword()) :: %{analyzed: non_neg_integer(), skipped: non_neg_integer()}
def analyze_all(opts \\ []) do
skip_recent = Keyword.get(opts, :skip_recent_seconds, 6 * 3600)
q = """
SELECT relname
FROM pg_stat_user_tables
WHERE schemaname = 'public'
AND (last_autoanalyze IS NULL
OR last_autoanalyze < now() - ($1 || ' seconds')::interval)
"""
%{rows: rows} = Repo.query!(q, [Integer.to_string(skip_recent)])
tables = Enum.map(rows, fn [n] -> n end)
Enum.each(tables, fn t ->
Repo.query!("ANALYZE #{quote_ident(t)}", [], timeout: :infinity)
end)
skipped =
Repo.query!("""
SELECT count(*) FROM pg_stat_user_tables
WHERE schemaname = 'public'
AND last_autoanalyze >= now() - ('#{skip_recent} seconds')::interval
""").rows
|> hd()
|> hd()
%{analyzed: length(tables), skipped: skipped}
end
# Defensive: pg_stat_user_tables returns legitimate identifiers from
# the catalog, but we still quote just in case a partition contains a
# character the shell would interpret.
defp quote_ident(name) when is_binary(name) do
escaped = String.replace(name, ~s("), ~s(""))
~s("#{escaped}")
end
@doc """
Derive `surface_refractivity` / `min_refractivity_gradient` /
`ducting_detected` for every `hrrr_profiles` row where those columns

View file

@ -0,0 +1,66 @@
defmodule Microwaveprop.Repo.Migrations.TuneAutovacuumForHighChurn do
use Ecto.Migration
@moduledoc """
Per-table autovacuum/analyze parameters for the hottest paths.
Default PG 16 thresholds (`scale_factor 0.2` for vacuum, `0.1` for
analyze) are too lazy for a handful of tables we hit frequently:
* `contacts` every enrichment tick UPDATEs one or more status
columns. 81 k rows × 0.1 analyze scale = 8 k dead tuples before
stats refresh; BackfillEnqueueWorker's status-priority ORDER BY
then picks a bad plan. Drop to 0.02 / 0.01.
* `grid_tasks` / `hrrr_fetch_tasks` small queue tables with a
claim/complete cycle that rewrites every row each hour. Defaults
mostly keep up, but the claim query's `FOR UPDATE SKIP LOCKED`
scan benefits from fresh stats. Set thresholds so vacuum/analyze
fire at 50 modified rows instead of ~(default 50) + 20%.
* `contact_common_volume_radar` MechanismClassifyWorker UPDATEs
bring the analyze stale-tuple counter to ~200 k without ever
flipping the default 10 % threshold. Matches the contacts
shape at 0.02 / 0.01.
All other tables keep PG defaults including `hrrr_profiles`
partitions (append-heavy, analyze-fires-naturally on insert) and
the Oban-owned tables (`oban_jobs`, `oban_producers`, `oban_peers`),
which Oban's own Pruner handles.
Reversible `down` resets each back to the cluster-wide default.
"""
@high_churn [
{"contacts", 0.02, 0.01},
{"grid_tasks", 0.05, 0.02},
{"hrrr_fetch_tasks", 0.05, 0.02},
{"contact_common_volume_radar", 0.02, 0.01}
]
def up do
for {table, vac_scale, analyze_scale} <- @high_churn do
execute("""
ALTER TABLE #{table} SET (
autovacuum_vacuum_scale_factor = #{vac_scale},
autovacuum_analyze_scale_factor = #{analyze_scale},
autovacuum_vacuum_threshold = 50,
autovacuum_analyze_threshold = 50
)
""")
end
end
def down do
for {table, _, _} <- @high_churn do
execute("""
ALTER TABLE #{table} RESET (
autovacuum_vacuum_scale_factor,
autovacuum_analyze_scale_factor,
autovacuum_vacuum_threshold,
autovacuum_analyze_threshold
)
""")
end
end
end