From 68354af021ff329e9e886872be071d900cc1468c Mon Sep 17 00:00:00 2001 From: Graham McIntire Date: Wed, 25 Mar 2026 12:36:10 -0500 Subject: [PATCH] fix: prevent Oban index bloat by reindexing primary key and scheduling index (#165) Severe index bloat discovered on oban_jobs: - oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste - oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec) causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space as reusable but cannot shrink indexes. The Oban Reindexer plugin was only reindexing args_index and meta_index by default, missing the primary key and main scheduling index. Solution: Explicitly configure Reindexer to include all high-churn indexes: - oban_jobs_pkey (primary key, used by all queries) - oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling) - oban_jobs_args_index (default, GIN index) - oban_jobs_meta_index (default, GIN index) Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation. Manual immediate reindex required to reclaim 11GB: REINDEX INDEX CONCURRENTLY oban_jobs_pkey; REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index; Reviewed-on: https://git.mcintire.me/graham/towerops-web/pulls/165 --- config/runtime.exs | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/config/runtime.exs b/config/runtime.exs index 9358b978..d970959c 100644 --- a/config/runtime.exs +++ b/config/runtime.exs @@ -261,8 +261,19 @@ if config_env() == :prod do # causing oban_jobs to bloat to millions of rows and slow all queries. # 50k/5s (10,000/s) keeps the table small. {Oban.Plugins.Pruner, max_age: 60, limit: 50_000, interval: to_timeout(second: 5)}, - # Rescue orphaned jobs (when node crashes) - {Oban.Plugins.Reindexer, schedule: "0 2 * * *"}, + # Rescue orphaned jobs (when node crashes) and prevent index bloat + # High DELETE churn (10k jobs/sec pruned) causes severe index bloat on: + # - Primary key (82.2 bloat, 3.2GB waste) + # - Scheduling index (216.8 bloat, 7.9GB waste) + # REINDEX CONCURRENTLY reclaims space without blocking operations. + {Oban.Plugins.Reindexer, + schedule: "0 2 * * *", + indexes: [ + "oban_jobs_pkey", + "oban_jobs_state_queue_priority_scheduled_at_id_index", + "oban_jobs_args_index", + "oban_jobs_meta_index" + ]}, # Rescue jobs stuck in executing state (mark as cancelled after 10 minutes) {Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 10)} ]