fix: prevent Oban index bloat by reindexing primary key and scheduling index (#165)

Severe index bloat discovered on oban_jobs:
- oban_jobs_pkey: 82.2 bloat ratio, 3.2GB waste
- oban_jobs_state_queue_priority_scheduled_at_id_index: 216.8 bloat, 7.9GB waste

Root cause: High DELETE churn from aggressive Pruner config (10k jobs/sec)
causes PostgreSQL VACUUM to leave dead index entries. VACUUM only marks space
as reusable but cannot shrink indexes.

The Oban Reindexer plugin was only reindexing args_index and meta_index by
default, missing the primary key and main scheduling index.

Solution: Explicitly configure Reindexer to include all high-churn indexes:
- oban_jobs_pkey (primary key, used by all queries)
- oban_jobs_state_queue_priority_scheduled_at_id_index (job scheduling)
- oban_jobs_args_index (default, GIN index)
- oban_jobs_meta_index (default, GIN index)

Daily REINDEX CONCURRENTLY at 2 AM will prevent future bloat accumulation.

Manual immediate reindex required to reclaim 11GB:
  REINDEX INDEX CONCURRENTLY oban_jobs_pkey;
  REINDEX INDEX CONCURRENTLY oban_jobs_state_queue_priority_scheduled_at_id_index;

Reviewed-on: graham/towerops-web#165
This commit is contained in:
Graham McIntire 2026-03-25 12:36:10 -05:00 committed by graham
parent 94fd8780de
commit 68354af021

View file

@ -261,8 +261,19 @@ if config_env() == :prod do
# causing oban_jobs to bloat to millions of rows and slow all queries.
# 50k/5s (10,000/s) keeps the table small.
{Oban.Plugins.Pruner, max_age: 60, limit: 50_000, interval: to_timeout(second: 5)},
# Rescue orphaned jobs (when node crashes)
{Oban.Plugins.Reindexer, schedule: "0 2 * * *"},
# Rescue orphaned jobs (when node crashes) and prevent index bloat
# High DELETE churn (10k jobs/sec pruned) causes severe index bloat on:
# - Primary key (82.2 bloat, 3.2GB waste)
# - Scheduling index (216.8 bloat, 7.9GB waste)
# REINDEX CONCURRENTLY reclaims space without blocking operations.
{Oban.Plugins.Reindexer,
schedule: "0 2 * * *",
indexes: [
"oban_jobs_pkey",
"oban_jobs_state_queue_priority_scheduled_at_id_index",
"oban_jobs_args_index",
"oban_jobs_meta_index"
]},
# Rescue jobs stuck in executing state (mark as cancelled after 10 minutes)
{Oban.Plugins.Lifeline, rescue_after: to_timeout(minute: 10)}
]